Research report: Vision based steering control using convolutional neural networks with latency analysis and microcontroller based actuation

A budget hardware combination of a host PC, an Arduino UNO microcontroller and a servo motor has been used to demonstrate a complete vision-based steering pipeline, in which a convolutional neural network predicts steering angles from camera images and physically actuates them through a servo motor. The study, published in Discover Robotics, reports a validation mean absolute error of 0.093 and a root mean square error of 0.131 on a widely used driving dataset, together with an estimated cumulative processing and actuation latency of approximately 150 milliseconds per frame, corresponding to roughly six to seven frames per second. The authors position the work explicitly as a hardware-validated proof of concept for low-speed operation rather than as a fully validated autonomous driving controller.

The neural network at the heart of the system is deliberately compact. It uses three sequential convolutional layers with 32, 64 and 128 filters respectively, each followed by max-pooling to progressively shrink the spatial dimensions of the feature maps and control computational load. Rectified linear unit activations follow each convolutional stage, and a 128-neuron fully connected layer, which alone accounts for approximately 5.97 million trainable parameters, maps the extracted spatial features to a continuous steering output. The authors note that the architecture was optimized primarily for real-time inference rather than for a minimum parameter count, representing a compromise between representational capacity and inference speed.

Each timestamped image is paired with a ground-truth steering angle, allowing supervised end-to-end learning of the mapping from visual input to steering command. Before training, all images were resized to 128 by 227 pixels, pixel intensities were normalized to the range zero to one, and data augmentation through horizontal flipping, random cropping and slight rotations of up to five degrees was applied to improve generalization. The data were split 80 percent for training and 20 percent for validation.

The model was trained with the Huber loss function, which combines the sensitivity of mean squared error to small deviations with the robustness of mean absolute error to outliers, making it suitable for regression tasks where occasional noisy labels may occur. Optimization used the Adam adaptive learning rate algorithm with an initial learning rate of 0.001, beta values of 0.9 and 0.999, a batch size of 64, and 50 epochs of training with early stopping after a patience of seven epochs to prevent overfitting. Training was performed on a system equipped with an Intel i7 CPU and an NVIDIA GPU. The training curve showed stable convergence, with the lowest validation mean squared error reaching 0.0172.

On the hardware side, the trained network runs on a host PC that processes frames from a USB camera, while the Arduino UNO receives each predicted steering command over serial communication and converts it into a pulse width modulation signal for an MG996R servo motor mechanically linked to the steering mechanism. The timing analysis breaks the pipeline into its sequential stages: camera capture contributes approximately 33 milliseconds, image preprocessing 7 milliseconds, CNN inference 70 milliseconds, serial communication 3 milliseconds, and servo actuation 37 milliseconds, for a cumulative budget of roughly 150 milliseconds per frame. The authors caution that this figure is an approximate budget rather than a statistically characterized worst-case or percentile latency, since the study did not use synchronized hardware timestamping or repeated latency trials, and effects such as camera buffering and operating-system scheduling were not independently isolated.

Given that latency, the study recommends a conservative operational boundary of below approximately 10 kilometres per hour for the current implementation. The authors stress that this boundary is a practical operating recommendation rather than an experimentally established vehicle-control limit, because vehicle-level lateral tracking was not quantitatively evaluated. At higher speeds, the same processing delay would introduce greater spatial displacement during the control delay and could contribute to phase lag and steering oscillations, particularly during rapid directional changes. As a possible mitigation, the paper suggests microcontroller-level interpolation between successive predicted steering angles, which could smooth frame-to-frame servo commands, although it would not eliminate the fundamental sensing and inference latency.

In benchmark comparisons under identical training, validation and testing splits, the proposed three-convolutional-layer model reduced the mean absolute error by approximately 27.3 percent relative to a shallow CNN baseline, from 0.128 to 0.093, while adding only about 30 milliseconds of inference time. The NVIDIA PilotNet architecture achieved a slightly lower error of 0.081 but required approximately 110 milliseconds of CNN inference, compared with roughly 70 milliseconds for the proposed model, a reduction of about 36 percent. The shallow CNN was faster at around 40 milliseconds but showed substantially higher prediction error. The authors emphasize that the reported inference times refer only to model execution on the host PC, whereas the 150-millisecond figure covers the full pipeline from image acquisition to servo response.

Beyond the labelled benchmark, the model was tested on external video sources, including publicly available driving footage from YouTube, gameplay from the driving simulation games Grand Theft Auto V and City Car Driving Simulator, and real-world rural road footage featuring uneven terrain and narrow roads. Because these external videos lacked synchronized ground-truth steering annotations, the evaluations were treated as qualitative assessments of visual-domain generalization rather than quantitative measures of accuracy. In the evaluated frames, predicted steering responses remained qualitatively consistent with the visible road geometry, although minor discrepancies were occasionally observed in scenarios with unfamiliar visual cues or complex road conditions. A graphical steering-direction indicator overlaid on the videos represented the model’s own output, not an independently measured steering angle.

The authors are careful to delineate what the results do and do not establish. The reported error metrics come from offline evaluation of the labelled validation set and can be influenced by the distribution of steering commands; a predominance of straight or mildly curved segments may yield low aggregate errors while saying little about performance during sharp turns. Moreover, the system operates as an open-loop command path in which the CNN’s prediction is transmitted directly to the servo, and although an MPU6050 inertial measurement unit was included in the hardware as a provision for future sensor-assisted feedback, its measurements were not incorporated into training, prediction, latency calculation or evaluation. Vehicle-level quantities such as lateral position error, heading error and closed-loop stability were not measured, and the term steering control in the paper refers to the generation and physical actuation of CNN-predicted commands rather than demonstrated trajectory control.

Within those limits, the study demonstrates that affordable, widely available components can support real-time neural steering-command generation and physical actuation, providing what the authors describe as a practical bridge between offline steering prediction and hardware actuator testing. They note that the current frame-wise approach lacks temporal awareness and may exhibit minor fluctuations during rapid directional transitions, and they identify temporal modelling, multisensor fusion and fully embedded edge deployment as future directions. Quantitative validation of autonomous driving performance, they conclude, would require controlled closed-loop vehicle experiments with synchronized measurements of steering, vehicle speed, lateral position and related vehicle-state parameters.

Subject of Research: CNN-based vision steering prediction with host-PC inference and Arduino servo actuation as a low-speed hardware proof of concept

Article Title: Vision based steering control using convolutional neural networks with latency analysis and microcontroller based actuation

Article References: Sharma, M. (2026). Vision based steering control using convolutional neural networks with latency analysis and microcontroller based actuation. Discover Robotics, 2(1), Article 31. https://doi.org/10.1007/s44430-026-00047-z

Image Credits: AI Generated

DOI: 10.1007/s44430-026-00047-z

Keywords: convolutional neural network, steering angle prediction, autonomous driving, Arduino UNO, servo actuation, latency, SullyChen dataset, PilotNet, Huber loss, proof of concept, computer vision, robotics