The research, conducted by Yijiang Zhang and Yanfeng Fang of the School of Artificial Intelligence at Shaoxing Institute of Technology in China, together with Wei Wang of the Department of Computer Engineering at the University of Rahjuyan Danesh Borazjan in Iran, introduces what the authors call a Reliability-Aware Adaptive Actor-Critic framework, abbreviated RACC. Its target is a core task in modern telecommunications known as service function chain (SFC) deployment, the process of stringing together a sequence of virtualized network functions, such as firewalls, load balancers, video optimizers, or encryption modules, so that traffic flows through them in the correct order while consuming as little of the edge’s limited computing capacity as possible.

The underlying technology that makes SFCs possible is network function virtualization, or NFV. Traditionally, network functions ran on dedicated hardware appliances, each one a physical box bolted into a rack. NFV replaces those boxes with software that can be instantiated on any sufficiently capable server, including the compact servers at MEC sites. That flexibility is enormously powerful, but it also creates a combinatorial explosion of choices. For every incoming service request, an operator must decide where to place each virtual network function (VNF), how much CPU, memory, and other resources to allocate to it, and how to route traffic between the functions. With dynamic service requests arriving continuously, heterogeneous network conditions, and limited edge resources, the search space is vast and constantly shifting.

Reinforcement learning has emerged as a natural fit for this kind of sequential decision-making problem. In a reinforcement learning setup, an agent learns by trial and error: it takes actions, observes the consequences, and adjusts its behavior to maximize a cumulative reward. Several prior approaches have applied reinforcement learning to SFC deployment, but the authors of the new study point out a persistent blind spot. Most existing methods concentrate on improving resource utilization and reducing latency while overlooking two critical dimensions: the reliability of the deployed service and the need for adaptive decision-making in highly dynamic MEC environments. A chain that is fast and efficient on paper is of little value if a node failure or resource bottleneck causes the service to degrade or fail outright.

RACC addresses this gap with an adaptive actor-critic architecture, a well-established family of reinforcement learning algorithms that separates the learning problem into two cooperating components. The actor is responsible for selecting actions, in this case the placement and resource allocation decisions for each service function chain, while the critic evaluates those actions by estimating their long-term value, providing feedback that helps the actor improve over time. The adaptive element allows the framework to adjust its behavior as conditions at the edge change, which is essential in an environment where traffic patterns, available capacity, and network states fluctuate from moment to moment.

What the framework optimizes is deliberately multi-dimensional. The authors formulate a multi-objective objective function designed to jointly maximize the SFC acceptance rate, meaning the proportion of incoming service requests the system can successfully accommodate, and service reliability, while simultaneously minimizing service latency, energy consumption, resource utilization, and migration cost. These goals frequently pull against one another. Accepting more requests strains resources and can raise latency; minimizing energy consumption may conflict with reliability guarantees; migrating functions to rebalance load costs time and disruption. By encoding all of these objectives into a single reward structure, RACC forces the learning agent to find deployment strategies that strike a principled balance rather than excelling on one metric at the expense of the others.

Two mechanisms inside RACC deserve particular attention. The first is dependency-aware SFC partitioning. A service function chain is not just a flat list of components; the functions have dependencies on one another, and the framework exploits this structure by partitioning chains into segments that can be executed in parallel. Parallel execution can shorten the end-to-end processing time of a chain, but only if the partitioning respects the dependency relationships between VNFs, otherwise the parallel segments would produce incorrect results or deadlock. By making the partitioning dependency-aware, RACC gains the speed benefits of parallelism without sacrificing correctness.

The second mechanism combines live migration with VNF reuse. Live migration allows a running virtual network function to be moved from one edge node to another without shutting it down, which the framework uses to maintain a balanced load distribution across the edge infrastructure and to enhance service reliability by relocating functions away from stressed or threatened nodes. VNF reuse, meanwhile, allows an already-running instance of a function to serve multiple service requests rather than spinning up a duplicate copy for each one. Together, these mechanisms improve resource allocation, keep load evenly distributed, and bolster the reliability of deployed services, all of which feed back into the reinforcement learning loop as the agent learns when and where to migrate and reuse.

The authors evaluated RACC through simulation against state-of-the-art baseline methods, including other reinforcement learning-based models. The results, as reported in the paper, show that the proposed framework consistently outperformed the baselines, and that RACC achieved higher expected cumulative reward values than the reinforcement learning baselines it was compared with. In reinforcement learning terms, a higher expected cumulative reward means the agent’s learned policy delivers better long-term outcomes across the full set of objectives, not just in isolated moments, which is precisely the property an operator needs when deployment decisions have cascading effects on future capacity and reliability.

The significance of this work extends beyond the simulation results themselves. As 5G and future 6G networks carry ever more latency-sensitive and resource-intensive applications, from industrial automation to immersive media, the pressure on edge infrastructure will only intensify, and the cost of ignoring reliability will grow with it. By demonstrating that a single adaptive actor-critic framework can jointly handle placement, resource allocation, parallel partitioning, migration, and reuse while explicitly accounting for reliability, the study offers a template for how the next generation of edge orchestrators might be built. The article is open access, published on 13 October 2026 in the Journal of Big Data, allowing researchers and network engineers alike to examine the full technical details of the framework.

Subject of Research: Reliability-aware reinforcement learning for service function chain deployment in mobile edge computing

Article Title: Adaptive actor-critic framework for efficient and reliability-aware SFC deployment in mobile edge computing

Article References: Adaptive actor-critic framework for efficient and reliability-aware SFC deployment in mobile edge computing. (n.d.). https://doi.org/10.1186/s40537-026-01580-9

Image Credits: AI Generated

DOI: 10.1186/s40537-026-01580-9

Keywords: mobile edge computing, network function virtualization, service function chain, reinforcement learning, actor-critic, reliability-aware deployment, VNF placement, resource allocation, live migration, latency optimization, energy consumption, Journal of Big Data

The motivation is straightforward. Muyu Song has been migrating from temple courtyards and oral transmission to short-video platforms and streaming channels, and while that shift has widened its potential audience, it has not automatically deepened its impact. The researchers point to three structural failures in current digital dissemination: content is fragmented in ways that ignore the music’s internal structure, audience feedback is scattered and slow to arrive, and publishing decisions rely on empirical rules that cannot adapt. The result is heritage content that reaches many people but often fails to hold them, engage them, or transmit the cultural meaning embedded in the performance.

What makes the new work unusual is its insistence that a folk song is not ordinary online content. Muyu Song performances contain flexible vocal pauses, beat-dependent chanting structures, dialectal expressions, rhyme-based filler words, and long narrative sequences. Generic AI pipelines slice audio into fixed-duration clips and run lyrics through standard Mandarin text encoders, which the authors show is a recipe for boundary errors and semantic dilution. Instead, their framework organizes every recording into segment-level samples whose boundaries are defined by silence, beat changes, chanting pauses, and narrative turns, so that each unit of dissemination corresponds to a coherent musical and narrative moment rather than an arbitrary time window.

Technically, the pipeline begins with a dual-branch audio encoder. A short-time Fourier transform produces a power spectrum that is mapped through a Mel filter bank into log-Mel acoustic features, while a parallel rhythm branch extracts beat period and phase information from onset intensities and tempograms. A lightweight convolutional network captures local spectral texture, a BiLSTM-Transformer combination encodes longer temporal structure, and attention pooling compresses everything into a segment-level audio embedding. On the text side, time-aligned lyrics are normalized so that function words, filler words, and rhythmic particles are annotated but preserved, with their temporal positions intact, before a Transformer encoder generates context-sensitive semantic vectors. A time-weighted attention pooling mechanism then aligns the text embedding with the audio embedding on the same timeline.

Fusion is handled by a gated cross-modal attention mechanism rather than simple concatenation. Audio features act as acoustic priors that reweight the text representation, giving more attention to lyric segments correlated with strong beats and emotional shifts, while textual semantics reweight the audio features to suppress background percussion and redundant chanting. The result is a single fused vector per segment that jointly encodes vocal style, rhythm, narrative, and dialect meaning. The gating weight is learned jointly from both modalities, so the balance between sound and sense shifts adaptively depending on the content of each segment.

On top of this representation sits a prediction model with a shared feature trunk and five separate regression heads, one each for click-through rate, first-completion rate, interaction intensity, share rate, and propagation depth. Crucially, the model also learns five cultural-quality variables: cultural understanding degree, cultural identity tendency, inheritance willingness, misinterpretation risk, and traditional-context fidelity, drawn from post-view questionnaires, semantic analysis of audience comments, and expert review of whether lyrics, dialect meanings, and performance context were correctly understood. These cultural indicators do not replace the traffic metrics; they act as constraint variables and penalty references, preventing the optimizer from degenerating into pure click-chasing.

The decision layer casts dissemination as a Markov decision process with a state combining the fused content features, platform conditions, and audience context, and an action space of 300 discrete configurations spanning four platforms, three presentation formats, five time windows, and five reach-frequency levels. A culturally constrained Actor-Critic optimizer learns a policy that maximizes a discounted long-term return, in which the weighted traffic reward is balanced against resource and disturbance costs and penalized whenever predicted cultural-quality indicators fall below predefined thresholds. Training uses epsilon-greedy exploration with the exploration rate annealed from 0.30 to 0.05, a replay buffer of 5,000 trajectories, mini-batches of 128 transitions, and an adaptive learning-rate rule that slows updates when the environment is stable and accelerates exploration when audience behavior or platform mechanisms drift.

Because the policy learns from predicted outcomes rather than always waiting for delayed platform analytics, the authors acknowledge the risk of the optimizer exploiting systematic errors in its own predictor. They counter this with three safeguards: the predictor is retrained every 500 newly observed outcomes, high-uncertainty predictions are down-weighted in the policy gradient using ensemble variance, and rewards are clipped to prevent the agent from chasing outlier estimates. The experiments themselves were run in a log-reconstructed environment built from real dissemination records rather than a live production system.

The dataset is substantial: 23,540 segment-level samples drawn from Muyu Song content published between January 2021 and December 2023 on a provincial intangible cultural heritage digital platform, cultural-institution media accounts, and public short-video and audio channels. Lyrics were transcribed by automatic speech recognition and manually corrected by annotators familiar with Muyu Song expressions, with dialect tags resolved through secondary review. To prevent data leakage, the 487 songs were split at the song level, so no segments from the same title appeared in both training and test partitions. All experiments were repeated five times with identical data partitions, seeds, and preprocessing.

The results give the domain-specific design choices concrete support. Compared with fixed-rule scheduling, the complete framework improved click-through rate by 8.2 percent, first-completion rate by 6.6 percent, interaction intensity by 11.2 percent, share rate by 9.5 percent, and propagation depth by 23.8 percent, and it also outperformed a generic multimodal Actor-Critic baseline. The ablations are the most telling part: replacing rhythm-aware boundaries with fixed windows reduced propagation depth to 1.87, removing dialect-preserving encoding lowered retention and sharing, and dropping the cultural constraints raised traffic slightly but pushed the misinterpretation risk rate from 0.073 to 0.126 while cutting traditional-context fidelity from 0.887 to 0.813. In other words, an algorithm that understands beats, dialect, and cultural meaning spreads the heritage further and distorts it less. The authors caution that performance still depends on sample size, platform data integrity, and feedback stability, and they flag cross-platform transferability and long-term cultural value as open questions, but the study makes a compelling case that keeping a folk tradition alive online may require teaching the machine to hear it the way its singers do.

Subject of Research: AI-based optimization of digital dissemination paths for Muyu Song intangible cultural heritage

Article Title: Research on optimizing the heritage transmission path of Muyu Song using AI technology

Article References: Chen, K., Li, F., & Chen, J. (2026). Research on optimizing the heritage transmission path of Muyu Song using AI technology. Discover Artificial Intelligence, 6(1), Article 1407. https://doi.org/10.1007/s44163-026-02304-x

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02304-x

Keywords: Muyu Song, intangible cultural heritage, reinforcement learning, multimodal fusion, Actor-Critic, Markov decision process, dissemination prediction, digital heritage, dialect encoding, rhythm-aware segmentation, cultural fidelity, short-video platforms

The study, conducted by Haotian Zhang, Chen Wang, and Jun Moon of the Department of Electrical Engineering at Hanyang University in Seoul, introduces TACCO, short for TD3+BC with Actor-Critic Contrastive Optimization. The method builds on TD3+BC, a minimalist and widely used offline reinforcement learning algorithm, and augments it with supervised contrastive learning branches that explicitly model out-of-distribution actions. The authors define out-of-distribution actions as those that lack local data support in a given state, do not match the current state, or exhibit low reliability. Rather than treating these problematic actions as an afterthought, TACCO makes them a first-class target of representation learning, reshaping the internal embeddings of the agent so that unreliable actions become visibly distinguishable from trustworthy ones.

To understand why this matters, it helps to see how offline reinforcement learning goes wrong. When an agent learns only from a fixed dataset, its value function, the critic that estimates the long-term reward of taking an action in a state, is only trustworthy for state-action pairs that actually appear in the data. During policy improvement, the actor proposes new actions that may drift outside the support of the dataset. The critic, having never seen these pairs, tends to overestimate their value, a phenomenon known as value overestimation. The actor then chases these inflated estimates, proposing ever more exotic actions, and the whole learning process spirals into degradation. This feedback loop between an overconfident critic and an adventurous actor is the central failure mode of offline reinforcement learning.

Existing approaches generally respond in one of two ways. Policy-constraint methods, such as TD3+BC itself, tether the learned policy to the behavior policy recorded in the data, penalizing any action that strays too far from what the dataset contains. Conservative value regularization methods, exemplified by CQL, pessimistically lower the estimated values of out-of-distribution actions so the critic never overhypes them. Both families work, but as the Hanyang team points out, few methods explicitly model the problematic actions from a representation learning perspective. That is the gap TACCO fills: instead of only constraining behavior or deflating values, it teaches the network’s internal representations to separate reliable actions from unreliable ones, providing a structural safeguard that complements the existing toolkit.

TACCO comes in two variants, each targeting a different side of the actor-critic architecture and a different reward regime. The first, TACCO-A, is designed for dense reward environments, where feedback signal is abundant. It attaches a supervised contrastive learning branch to the actor. The actor consists of a shared state-encoding backbone, a standard policy output head, and an additional embedding head that concatenates the hidden state representation with the proposed action and projects the result into a 64-dimensional embedding. Contrastive learning then operates on three kinds of actions: data-driven actions drawn from the offline dataset, random actions, and cross-state policy actions. The training objective pulls embeddings of dataset-supported actions together while pushing embeddings of random and mismatched actions apart, so the actor’s representation space develops a clear geometry in which out-of-distribution actions stand out and can be suppressed.

The second variant, TACCO-B, is tailored to sparse reward environments, where rewards arrive rarely and unreliable value estimates are especially corrosive. Here the contrastive branch moves to the critic. TACCO-B filters training samples on the critic side based on two criteria: high Q-values and low uncertainty. Samples that pass both filters are treated as reliable, while the remainder are flagged as suspect. The method then applies contrastive constraints to both reliable and remaining samples simultaneously, encouraging the critic’s shared state-action encoder, a small network ending in a LayerNorm-normalized 64-dimensional embedding, to learn discriminative representations that reflect this reliability structure. By shaping the critic’s representation space rather than merely clipping its outputs, TACCO-B aims to stabilize value learning even when the reward signal is thin and misleading bootstrapped targets are easy to come by.

The architectural details are deliberately modest, which is part of the method’s appeal. Both variants retain the dual Q-network structure of TD3, with two independent critics that help mitigate function approximation error, and both keep the behavioral cloning term of TD3+BC that anchors the policy to the dataset. The contrastive modules are small multilayer perceptrons, and the authors report that performance is relatively robust to the weighting of the contrastive loss. Sensitivity analyses reveal a more interesting story for the other knobs: in TACCO-A, relaxing the behavioral cloning weight makes the policy more adventurous but destabilizes training, and increasing the contrastive weight steadily compensates for that instability. In TACCO-B, the fraction of high-value samples used for filtering must be tuned carefully, since too few samples starve the model of coverage while too many reintroduce the very low-quality actions the filter was meant to exclude.

The empirical evaluation is broad. The team compared TACCO against six state-of-the-art offline reinforcement learning algorithms, including CQL, TD3+BC, the Decision Transformer, IQL, DTQL, and O-DICE, across 27 tasks spanning the MuJoCo locomotion suite and the Adroit dexterous manipulation suite within the D4RL benchmark. All methods were trained under an identical protocol: one million gradient steps, evaluation every five thousand steps, a batch size of 256, and results averaged over five random seeds with 95 percent confidence intervals, using the standard D4RL score normalization. According to the authors, TACCO achieved superior performance on most of these tasks, which they interpret as validation that mitigating out-of-distribution action interference through contrastive optimization enhances policy stability, in both dense and sparse reward settings.

The broader significance of the work lies in its reframing of a classic problem. Distribution shift has usually been handled as a constraint or a penalty, a blunt instrument applied to actions or values. TACCO instead treats it as a representation problem: if the network’s embeddings make the boundary between supported and unsupported actions explicit, then both the actor and the critic can exploit that structure during training. This connects offline reinforcement learning to the wider wave of contrastive learning that has transformed vision and speech processing, and to a growing line of research that uses contrastive objectives for goal-conditioned control and planning. It also suggests a practical path forward, since the contrastive branches add little architectural complexity on top of a minimalist baseline that many practitioners already deploy.

Challenges remain before such methods reach real-world robotics and industrial control, where offline data is often suboptimal, heterogeneous, and collected under shifting conditions. The authors note that code will be made available on request, and the work was supported by Korean government research programs through KETEP, MOTIE, and IITP. As offline reinforcement learning matures into a serious option for settings where online exploration is prohibitive, techniques like TACCO point toward a hybrid future: conservative enough to trust the data, expressive enough to improve on it, and, crucially, aware of exactly where its own knowledge ends.

Subject of Research: Contrastive actor-critic optimization for mitigating out-of-distribution actions in offline reinforcement learning

Article Title: TACCO: Actor–Critic Contrastive Optimization for OOD Action Mitigation in Offline Reinforcement Learning

Article References: Zhang, H., Wang, C., & Moon, J. (2026). TACCO: Actor–Critic Contrastive Optimization for OOD Action Mitigation in Offline Reinforcement Learning. Machine Learning, 115(10), Article 241. https://doi.org/10.1007/s10994-026-07170-3

Image Credits: AI Generated

DOI: 10.1007/s10994-026-07170-3

Keywords: offline reinforcement learning, contrastive learning, out-of-distribution actions, TD3+BC, actor-critic, D4RL benchmark, value overestimation, distribution shift, representation learning, MuJoCo, Adroit, machine learning

The core challenge the researchers set out to tame is one of the hardest in edge computing. Wireless channels fluctuate constantly as signals are blocked, interfered with, and faded. Task arrivals are random and unpredictable. A device’s battery drains as it computes or transmits. If tasks arrive faster than the system can process them, the queues of waiting data at the edge server grow without bound, and the whole system becomes unstable. Traditional optimization methods struggle in this environment because they typically assume knowledge of future conditions that simply does not exist. Machine learning approaches, particularly deep reinforcement learning, can adapt to changing conditions, but they often fail to guarantee the long-term stability of the system’s queues and its average power consumption, which are the metrics that determine whether a network remains usable over hours and days rather than seconds.

RIASO attacks this problem by fusing three techniques that have rarely been combined so tightly. The first is Lyapunov optimization, a mathematical framework that converts long-term stability requirements into per-time-frame decisions. Instead of trying to plan far into an uncertain future, Lyapunov optimization maintains virtual queues that track accumulated data backlog and power consumption, and then makes each frame’s decision in a way that pushes those queues back toward zero. This provides a theoretical guarantee of long-term stability without needing to predict the future. The second ingredient is deep reinforcement learning, in which a neural network learns through trial and error which offloading decisions yield the best outcomes. The third, and arguably the most novel, is multi-output learning, which trains a single neural network to produce many candidate solutions simultaneously rather than just one.

The architecture that ties these together is an actor-critic structure, a well-established reinforcement learning pattern in which one component proposes actions and another evaluates them. In RIASO, the actor is a multi-input multi-output convolutional neural network, dubbed MIMO_CNN, that receives three pieces of information for every user device at each time frame: the current wireless channel gain, the length of the device’s data queue, and the length of its virtual power queue. From this input, the network generates not one but multiple candidate binary offloading decisions, each specifying which devices should compute locally and which should offload to the edge server. The critic then evaluates each candidate analytically, solving the remaining resource allocation problem, which is convex once the binary decisions are fixed, using an efficient primal-dual algorithm. The best action is selected, stored in a replay memory, and used to retrain the actor network.

This multi-output design is where RIASO departs most sharply from conventional actor-critic schemes, which typically use a single-output network and rely on random exploration to discover good actions. By generating many diverse candidate solutions in parallel, the MIMO_CNN naturally balances exploration and exploitation: the system can pick the best of many well-informed options rather than gambling on random perturbations of a single guess. The researchers report that this leads to more robust and faster convergence of the training process, which matters enormously in edge networks with high-dimensional action spaces where the number of possible offloading combinations grows exponentially with the number of devices.

Equally inventive is the way RIASO handles its own training schedule. In standard deep reinforcement learning, the model is retrained every time new experience accumulates, which is computationally expensive and can slow the system down. RIASO instead waits until its replay memory is completely full and then trains on the entire batch at once, setting the batch size equal to the memory size so that every stored sample is used and the memory is emptied. Moreover, training is triggered only if the current loss exceeds the minimum of all previous loss values, preventing wasteful retraining when the model is already performing well. The researchers found that this policy cut the training phase from roughly 10,000 time frames down to just 2,000, a fivefold reduction that directly translates into shorter response times and shorter data queues for users.

The team also carefully tuned the network’s structure. They examined how the number of output branches in the MIMO_CNN affects performance, discovering that with only five output layers the system behaved erratically, with unreasonable queue lengths, while configurations with ten or more layers converged reliably. After 2,000 time frames, the average data queue lengths for ten, twenty, and thirty output layers were nearly identical, so the researchers settled on ten as the sweet spot between convergence quality and computational cost. Similarly, they swept the size of the replay memory and sample batch, finding that a size of 256 offered the best balance: smaller memories risked trapping the model in local optima, while larger ones updated the data too slowly and lengthened training time.

To validate RIASO, the researchers ran simulations on a TensorFlow 2.0 platform modeling a multi-user MEC network in which an access point integrated with an edge server assists a set of user devices whose tasks arrive stochastically across consecutive time frames. Channel gains followed a Rician distribution built on a path-loss model, with devices positioned at varying distances from the server. The benchmarks included LyDROO, a well-known Lyapunov-guided deep reinforcement learning framework; DRL-DO, a distributed offloading framework using multiple convolutional networks; and three naive strategies: executing everything locally, executing everything at the edge, and executing randomly. Across the weighted sum computation rate, average data queue length, average power consumption, average energy queue length, and average response time per channel, RIASO delivered the best computation rate and the shortest data queue of all tested methods.

The comparison with LyDROO was particularly instructive. Under identical conditions and parameters, RIASO performed comparably on most metrics but achieved better results on average response time per channel, average data queue length, and average energy queue length, indicating a more stable system overall. DRL-DO showed better average power consumption than RIASO, but at the cost of instability across the other metrics, which the authors argue is a fatal flaw for real deployments where queue blowups mean dropped tasks and angry users. The naive strategies illustrated the trade-offs vividly: local execution minimized response time but produced a low computation rate and high energy queues, while edge execution minimized power consumption but created very long data queues, and random execution was poor on nearly every measure.

The implications reach well beyond the simulation lab. As 5G and future 6G networks push computing closer to users, the offloading decision layer becomes the nervous system of the entire edge infrastructure, and algorithms like RIASO demonstrate that it is possible to be simultaneously fast, energy-aware, and provably stable without clairvoyant knowledge of the future. The authors point to several directions for extending the work, including multitier offloading across edge, fog, and cloud layers, collaborative offloading among neighboring devices with spare resources, and support for partial offloading in which a task’s data is split between local and remote execution. For now, RIASO stands as a compelling proof that combining Lyapunov optimization’s stability guarantees with the adaptive intelligence of multi-output deep reinforcement learning can produce an offloading algorithm that is greater than the sum of its parts, one that keeps the edge responsive even as the wireless world around it refuses to sit still.

Subject of Research: Real-time adaptive computation offloading in multi-access edge computing using Lyapunov optimization and deep reinforcement learning

Article Title: A novel real-time integrated adaptive stable offloading (RIASO) algorithm in multi-access edge computing

Article References: Elhadidy, H., Saleh, H., Rizk, R., & Saber, W. (2026). A novel real-time integrated adaptive stable offloading (RIASO) algorithm in multi-access edge computing. Cluster Computing, 29(14), Article 796. https://doi.org/10.1007/s10586-026-06444-8

Image Credits: AI Generated

DOI: 10.1007/s10586-026-06444-8

Keywords: edge computing, mobile edge computing, computation offloading, deep reinforcement learning, Lyapunov optimization, multi-output learning, Internet of Things, queue stability, actor-critic, convolutional neural network, energy efficiency, wireless networks

The research addresses a class of systems that engineers find particularly awkward: unknown, non-affine, discrete-time systems. In plain terms, these are processes sampled at fixed time steps, whose mathematical equations are either unknown or too complicated to write down, and in which the control input does not appear in a convenient, separable form. Most industrial equipment behaves this way in practice. Friction, saturation, and nonlinear electrical characteristics mean that the tidy models used in textbooks rarely match the hardware on the factory floor. Rather than trying to identify the system’s equations, the new approach is model-free: it learns directly from the data the system produces, adjusting its behavior online as measurements arrive.

The central insight of the work is that safety cannot be judged by the output alone. When a DC motor is commanded to follow a position trajectory, the variable that actually determines whether the machine survives is the motor current, an internal state that depends on the output and on the control action. If the controller drives the position aggressively, the current can spike, overheating the windings or tripping protective hardware. The study therefore treats constraints on these dependent internal states as first-class citizens in the control design, explicitly incorporating them into the learning objective rather than hoping that good output tracking will keep them in check as a side effect.

To achieve this, the author built the controller around a fuzzy-rule emulated actor-critic architecture. Actor-critic methods are a staple of reinforcement learning: one network, the actor, proposes control actions, while a second network, the critic, estimates how good those actions are and guides the actor toward better ones. The novelty here lies in the structure of these networks. Instead of conventional neural layers, they are built from fuzzy rules, which partition the input space into overlapping regions and respond with locally tuned outputs. This gives the architecture a piecewise, interpretable character that suits systems sampled in discrete time, and it allows the learning laws to be derived with clear analytical guarantees rather than relying purely on gradient heuristics.

Safety enters through a barrier function embedded in a hierarchical reward function. Barrier functions are a classical control-theory tool: they grow without bound as a state variable approaches a forbidden boundary, so any strategy that minimizes them is naturally repelled from unsafe regions. By folding such a barrier into a layered reward, the scheme makes the agent feel escalating penalties as the motor current nears its limits, while the upper levels of the hierarchy reward accurate tracking of the commanded position. The result is a controller that balances two competing goals in a principled way: it wants to follow the reference signal, but not at the cost of violating the constraints that keep the hardware healthy.

Learning in the scheme is carried out by two distinct online laws, each employing time-varying learning rates. Rather than adjusting the actor and critic with fixed step sizes, the algorithm modulates how aggressively it adapts depending on the operating conditions, which helps the networks converge quickly at the start of a task and then settle into stable, fine-grained refinement. The paper goes beyond simulation and empirical tuning: the effectiveness of these learning laws is theoretically demonstrated, with analysis showing that they ensure robust closed-loop performance. In the control community, where reinforcement learning proposals are often criticized for lacking formal guarantees, this combination of online adaptability and theoretical backing is a meaningful contribution.

The experimental validation is refreshingly concrete. Rather than testing only on numerical benchmarks, the author implemented the method on a real DC motor position control system, exactly the kind of electromechanical plant found in robotics, manufacturing lines, and automation equipment across industry. The results showed that the controller achieved sufficient tracking of the position reference while producing a significant reduction in the accumulating reward function compared with alternative approaches. In reinforcement learning terms, a lower accumulated reward cost means the controller found a way to do the job while incurring far less penalty, which here translates directly into gentler, safer behavior of the physical machine.

One of the most striking findings comes from an information-theoretic lens. The study compared the spectral entropy of the motor current, the constrained internal state, under the proposed scheme and under competing controllers. Spectral entropy measures how spread out and unpredictable a signal’s frequency content is. Under the new method, the spectral entropy of the current was significantly lower, and the probability distribution of the current was noticeably narrower. In practical terms, the motor current stayed in a tight, predictable band instead of wandering across a wide range of values. Narrow, low-entropy currents mean less thermal stress, less electrical noise, and smoother operation, which is precisely what a safety-oriented controller should deliver.

The implications extend well beyond a single motor on a laboratory bench. Model-free safe control of unknown discrete-time systems is a recurring need in fields as varied as wind turbine generators, electro-hydraulic servo systems, permanent magnet motor drives, and chemical process control, all of which appear in the paper’s extensive engagement with recent literature on adaptive dynamic programming, control barrier functions, and safe reinforcement learning. Many of those approaches either require a system model, handle constraints only on the input or the output, or offer no formal learning guarantees. By combining fuzzy-rule-based function approximation, hierarchical reward design, barrier-function safety, and provable online learning laws in a single model-free framework, the study offers a template that could be adapted to any plant where an observable output hides safety-critical internal states.

There are, of course, the usual caveats that accompany any single-author experimental study. The validation platform is a DC motor position control system, and extending the guarantees to higher-dimensional, multi-input systems with more complex constraint structures will require further work. The published article does not report external funding, and the author declares no conflict of interest. Still, the core message is likely to resonate widely: reinforcement learning controllers can be made genuinely safe, not by restricting them so heavily that they stop learning, but by building the physics of the constraints into the reward they optimize. As machines increasingly learn their own control policies online, approaches like this one, which watch the hidden variables that keep hardware alive, may become the standard grammar of trustworthy autonomous control.

Subject of Research: Model-free hierarchical reinforcement learning for safe output tracking with internal state constraints in unknown discrete-time systems

Article Title: Hierarchical reinforcement learning for safe output tracking: managing constraints of dependent internal states in unknown discrete-time systems

Article References: Treesatayapun, C. (2026). Hierarchical reinforcement learning for safe output tracking: managing constraints of dependent internal states in unknown discrete-time systems. Neural Computing and Applications, 38(18), Article 739. https://doi.org/10.1007/s00521-026-12456-7

Image Credits: AI Generated

DOI: 10.1007/s00521-026-12456-7

Keywords: reinforcement learning, safe control, actor-critic, fuzzy rules, barrier function, discrete-time systems, output tracking, DC motor control, state constraints, spectral entropy, model-free control, adaptive control