To present the overall linkage relationship between collaborative recovery methods in disaster disturbance driving, state construction, action generation, value evaluation, and strategy execution, a method system architecture diagram is introduced. Figure 1 shows the mapping process from multi-source disturbance inputs to high-dimensional state representation, revealing the coupling link between action space and physical consistency constraints, and demonstrating the parameter dependency relationship between the central evaluator and the electrical dual agent strategy network, forming the logical channel of the collaborative recovery mechanism.
Fig. 1
Overall framework of collaborative recovery method system.
The structure of Fig. 1 reflects the system offset driven by disaster disturbance, multi-scale encoding of state representation, mapping relationship between continuous and discrete actions, state transition supported by physical equations, feedback loop of reward signals, and training dependence between the central evaluator and the dual agents. In the overall framework, the three elements of state, action, and value form a closed-loop decision chain. The electrical subsystem strategy maintains consistent recovery logic under coupling constraints, and the temporal structure of the recovery path is guided by multi-objective rewards, resulting in a coordinated response to the cross network recovery process.
Construction of system state representation driven by disasters
Under extreme disaster conditions, the operating states of both the power system and the natural gas system deviate from the steady-state range, and there are significant differences in the dimensions, magnitude of changes, and response time scales of different physical quantities. In order for reinforcement learning strategies to obtain a unified perception of the overall damaged state of the system during the post disaster recovery phase, it is necessary to map multi-source operational information to the same continuous state space. The voltage amplitude, pressure distribution, branch current, node fault status, and resource constraints collected in disaster scenarios are uniformly encoded to form a continuous state vector. The node voltage of the power system is expressed as a vector \(\:\text{V}\in{\text{R}}^{{\text{N}}_{\text{e}}}\); the gas network pressure is expressed as a vector \(\:\text{P}\in{\text{R}}^{{\text{N}}_{\text{g}}}\); and the node fault state is expressed as a binary vector \(\:\text{F}{\epsilon{0,1}}^{N}_{e+N_g}\) expression.
The key characteristic of the system state under disaster action is not the absolute operating value, but the degree of deviation relative to the pre-disaster steady state. To explicitly characterize the operational offset caused by disaster disturbances, a differential description of operational variables is introduced29:
$$\:{\Delta}{\text{x}}_{\text{t}}\text{=}{\text{x}}_{\text{t}}\text{-}{\text{x}}_{\text{ref}}$$
(1)
In Eq. 1, \(\:{\text{x}}_{\text{t}}\) is the system operating quantity at time t, and \(\:{\text{x}}_{\text{ref}}\) is the reference steady-state quantity. Through differential modeling, the functional degradation caused by disasters is directly mapped as state shift, which is beneficial for establishing a consistent correlation between subsequent recovery goals and steady-state regression goals.
Due to significant differences in physical quantities such as voltage, pressure, and current at the numerical scale, direct concatenation can lead to scale dominance in the state space during the training process. To avoid interference from different dimensions on strategy learning, the differential components are further normalized. The normalization factor is composed of the node rated voltage, rated pressure, and maximum power flow of the branch, forming a vector s. Execution:
$$\:{\widehat{\text{x}}}_{\text{t}}\text{=}{\text{s}}^{\text{-1}}{\odot}{\Delta}{\text{x}}_{\text{t}}$$
(2)
In Eq. 2, ⊙ represents element wise multiplication and is used to handle dimensional differences between different physical quantities.
The process of disaster recovery has obvious temporal continuity, and the state at a single moment is difficult to reflect the evolutionary trend of the system from damage to recovery. To preserve the temporal characteristics of disaster disturbances, the normalized states are stacked and generated within the time window L30, allowing successive states to encode the delayed influence of pipeline storage release and pressure evolution:
$$\:{\text{H}}_{\text{t}}\text{=[}{\widehat{\text{x}}}_{\text{t}\text{-}\text{L}\text{+1}}{,\dots,}{\widehat{\text{x}}}_{\text{t}}\text{]}$$
(3)
In Eq. 3, \(\:{\text{H}}_{\text{t}}\) represents the trajectory of changes in various quantities on a continuous time scale under the influence of disasters, providing a more correlated input for the strategy network during the training phase.
In the electric–gas coupled coupling system, the voltage and pressure changes at the coupling nodes are not independent of each other, and their recovery order directly affects the energy mutual benefit effect across the system. To explicitly reflect this cross network association in the state space, a cross network mapping matrix \(\:\text{M}\in{\text{R}}^{\left({\text{N}}_{\text{e}}\text{+}{\text{N}}_{\text{g}}{)\times(}{\text{N}}_{\text{e}}\text{+}{\text{N}}_{\text{g}}\right)}\) is introduced to jointly map the temporal states:
$$\:{\text{Z}}_{\text{t}}\text{=}\text{M}{\text{H}}_{\text{t}}$$
(4)
In Eq. 4, \(\:{\text{Z}}_{\text{t}}\) forms a feature aggregation related to voltage and pressure in the representation space, which enhances the sensitivity of the intelligent agent to the recovery action of the coupled nodes during the training process. The sparse structure of matrix M is constructed based on the position of the electric drive compression equipment, the position of the gas drive power generation unit, and the topological coupling relationship, so that cross network mapping does not damage the integrity of the original system structure31,32.
Post disaster recovery decisions are not only constrained by physical conditions, but also directly constrained by the quantity and progress of repair resources. To synchronize the perception of resource feasibility during decision-making, a recovery resource vector \(\:{\text{R}}_{\text{t}}\) is introduced to record the amount of available resources, repair queue status, and remaining repair time. The availability state of coupling devices is synchronously encoded in this vector, representing binary availability and remaining repair time, and directly conditions whether the fixed-ratio energy conversion relation is activated in the subsequent state transition. After normalization, this vector is concatenated with the aforementioned state representation to form the final state input:
$$\:{\text{S}}_{\text{t}}\text{=[}{\text{Z}}_{\text{t}}\text{,}{\text{R}}_{\text{t}}\text{,}\text{F}\text{]}$$
(5)
In Eq. 5, St constitutes the complete state description of the reinforcement learning agent during the disaster recovery phase, enabling the strategy to simultaneously balance system operation offset, cross network coupling relationships, and resource constraints in a unified state space.
Design of recovery action space for electric gas subsystem
The decision-making goal in the post disaster recovery phase is to gradually restore system functionality through limited control actions, but there are differences between the power system and the gas grid in terms of controlling variable types and regulating continuity. To unify the description of different types of recovery operations, the adjustment of power grid switch status, gas grid valve adjustment, output changes of electric drive compressors, and output scheduling of gas drive power generation units are taken as the basic control variables, and the action vector \(\:{\text{A}}_{\text{t}}\) is formed through discrete and continuous joint coding.
The discrete action indices output by the reinforcement learning strategy network during the training phase correspond to predefined recovery control primitives and do not inherently satisfy the physical feasibility requirements of the electric gas system. If executed directly, it can result in state transition failure or violation of operational safety conditions. Therefore, before applying actions, unconstrained actions need to be projected onto the physical feasible space through feasible domain mapping33,34. Let the continuous control primitive associated with the selected discrete action index be denoted as \(\:{\text{A}}_{\text{t}}^{\text{u}}\in{\text{R}}^{\text{d}}\), and the feasible domain center vector c and scale matrix D are used to perform bounded mapping:
$$\:{\text{A}}_{\text{t}}\text{=}\text{c}\text{+}\text{D}{\text{A}}_{\text{t}}^{\text{u}}$$
(6)
In Eq. 6, \(\:{\text{A}}_{\text{t}}\) forms a bounded recovery control input that satisfies equipment operating ranges, while power balance and pressure balance are enforced through the subsequent coupled power flow and gas flow equation solving process. The diagonal elements of the scale matrix D are constructed based on the admissible operating envelope of coupling devices and network components, reflecting rated capacity boundaries and inter-step adjustment limits of compressors and gas-driven generators, so the mapped control input satisfies amplitude and ramping feasibility before entering the physical solver35.
The action vector is applied to generate the next time estimate of the system state through coupled power flow and gas flow equation solving. If the mapped continuous recovery action leads to non convergence or violation of physical admissibility conditions in either power flow or gas flow equations, the action is identified as constraint-rejected36. In this case, the physical state remains unchanged across the recovery step, and the environment returns a negative payoff reflecting infeasible control execution while the recovery timeline advances normally. The power system adopts the power flow equation \(\:{\text{g}}_{\text{e}}\), and the gas network adopts a steady-state flow equation \(\:{\text{g}}_{\text{g}}\) embedded in a discrete-time evolution structure, which together form a nonlinear state transition relationship describing the inter-step redistribution of electrical and gas states:
$$\:{\text{x}}_{\text{t}\text{+1}}\text{=}\text{f}\text{(}{\text{x}}_{\text{t}}\text{,}{\text{A}}_{\text{t}}\text{)}$$
(7)
In Eq. 7, f is defined by the coupled solution of power balance equations, gas flow equations, and fixed-coefficient energy conversion relations of coupling devices, where the admissible electrical and gas-side injections are determined by constant energy conversion coefficients, operating limits, and current availability states. For constraint-rejected actions, the state transition degenerates to an identity mapping, and the next state is set equal to the current state to reflect the absence of physical evolution under infeasible control input.
At the coupling node, the recovery of gas flow can directly affect the regulation capability available for power side recovery. To characterize this relationship, a coupling mapping operator is introduced to map the gas-side input into electrical-side power output under fixed energy conversion coefficients and operating limits of coupling devices:
$$\:{\text{P}}_{\text{t}}^{\text{c}}{=\eta}\text{}{\text{z}}_{\text{t}}$$
(8)
In Eq. 8, \(\:{\text{P}}_{\text{t}}^{\text{c}}\) represents the adjustable power at the coupling point; \(\:{\text{z}}_{\text{t}}\) is the gas-side energy input quantity; \(\:\eta\) denotes the fixed energy conversion coefficient of the coupling device, which defines a constant proportional relationship between gas consumption and electrical output under admissible operating conditions. The action is propagated through the physical constraint model to form the next physical state prediction, where system nonlinearity arises from network flow equations and operating constraints, while the coupling devices are represented by fixed proportional energy conversion relationships, and the resulting state transition is used for constructing the reinforcement learning objective function37.
To adapt to the reinforcement learning strategy structure, the recovery control space is parameterized by a finite discrete basis set, where each discrete index uniquely corresponds to a continuous recovery control primitive. The action discrete basis vector set is denoted as \(B=b_{1,b_2},\dots,b_M\), which is generated through the minimum adjustment step size, direction constraint, and topological neighborhood structure of adjustable devices38. Each basis vector defines a deterministic continuous control template acting on switch states, valve states, and coupling device outputs, and serves as the unique continuous representation of the selected discrete action. Continuous actions are projected onto this set using the minimum distance mapping: \(\:{\text{k}}_{\text{t}}\text{=}\text{arg}\underset{\text{i}}{\text{min}}\parallel{\text{A}}_{\text{t}}\text{-}{\text{b}}_{\text{i}}{\parallel}_{\text{2}}\), where the action number \(\:{\text{k}}_{\text{t}}\) is a discrete action that the agent directly selects, and the corresponding basis vector is deterministically restored as the continuous recovery control input. The numbering space is divided into the electrical system domain \(\:{\text{K}}_{\text{e}}\) and the gas system domain \(\:{\text{K}}_{\text{g}}\), and the numbering intervals do not overlap with each other. The shielding matrix of the coupled devices ensures that physically mutually exclusive actions are not selected39.
In the execution phase, the discrete number is restored to a continuous basis vector through inverse mapping: \(\:{\stackrel{{\sim}}{\text{A}}}_{\text{t}}\text{=}{\text{b}}_{{\text{k}}_{\text{t}}}\), and enters the physical constraint chain defined by Eq. 6 to 8 to ensure that the actual execution action satisfies power flow balance, pressure balance, and fixed-ratio energy conversion relationships, achieving a unified structure of discrete strategy and continuous control.
The effectiveness of the recovery action is quantified through a single step payoff function, which is constructed from voltage offset, pressure offset, and repair progress. When a recovery action is identified as constraint-rejected, the payoff function is overridden by a fixed negative penalty associated with infeasible execution, and no state improvement term is accumulated for that step. The system offset is expressed as a vector \({\Delta}{\text{x}}_{\text{t}\text{+}\text{1}}\), and the payoff function adopts a weighted two norm form40:
$$\:{\text{r}}_{\text{t}}{=-\Delta}{\text{x}}_{\text{t}\text{+1}}^{{\tau}}\text{W}{\Delta}{\text{x}}_{\text{t}\text{+1}}\text{-}{\rho}{{\delta}}_{\text{t}}$$
(9)
In Eq. 9, W is the state offset weight matrix; \(\:{{\delta}}_{\text{t}}\) denotes the consumption of repair resources under admissible control execution; and the payoff definition implicitly incorporates a fixed negative penalty when the applied action is classified as constraint-rejected, ensuring that infeasible actions are consistently discouraged during policy optimization. The payoff function is used during the training phase to update the gradient of the strategy network, ensuring that the recovery action continues to drive the system towards steady-state convergence.
The intelligent agent strategy network outputs a categorical probability distribution over the discrete recovery action space conditioned on the observed environmental state. By introducing disaster disturbance intensity as a regulating factor, the action space exhibits significant evolutionary characteristics under different degrees of damage. Figure 2 shows the evolution of action distribution with changes in disaster disturbance.
Fig. 2
Schematic diagram of action distribution changing with disaster disturbance. Note: P-Switch represents the action of power system switches, G-Valve represents the action of gas network valves, and an increase in index number represents the evolution of action attributes from conventional operation regulation to critical topology reconstruction.
The strategy distribution dynamically shifts with the increase of environmental disturbance intensity. In a low disturbance environment, the center of gravity of the action is maintained within the regular scheduling interval. As the disaster level increases, the high probability action area migrates to the key topology reconstruction action, reflecting the collaborative response characteristics of the electric gas subsystem under extreme working conditions. This mapping mechanism from local adjustment to global reconstruction ensures the adaptability of the recovery strategy in the multidimensional state space, achieving deep matching between damaged grid structures and energy supply recovery needs.
Construction of centralized training and distributed execution architecture
In the integrated energy system of electricity gas coupling, post disaster recovery decision-making involves multiple subsystems, whose recovery actions are mutually constrained at the physical level, but belong to different regulatory entities at the execution level. If centralized decision-making is fully adopted, it leads to a sharp expansion of the state dimension and action space, reducing the real-time response capability in disaster scenarios; if relying solely on distributed independent decision-making, it is difficult to characterize the relationship between cross system recovery sequence and energy reciprocity. Based on the above contradictions, a centralized training and distributed execution architecture is introduced to strike a balance between collaborative consistency and execution flexibility. The proposed framework is built upon the multi-agent gradient off-policy actor–critic algorithm introduced in Ren42, where the centralized critic evaluates joint state–action pairs and decentralized actors parameterize discrete recovery policies. The distinction from standard CTDE formulations lies not in the actor–critic structure itself but in three aspects specific to the electric–gas recovery problem. The centralized critic receives a joint state constructed from cross-network physical coupling variables rather than a concatenation of independent subsystem observations, so that the value estimate reflects the physical interdependence between voltage recovery and pressure restoration at coupling nodes. The action space is organized around a discrete basis set derived from the physical topology and coupling constraints of the integrated system, rather than a generic flattened action vector, ensuring that the critic evaluates actions whose physical meaning is consistent across training scenarios. The reward signal incorporates a delay propagation mechanism calibrated to the temporal lag between repair execution and observable system-level recovery, which is absent in standard actor–critic formulations designed for single-step or short-horizon reward structures. The loss functions and parameter update rules follow Ren42 and are not repeated here. The proposed architecture is instantiated as a centralized-critic multi-agent proximal policy optimization framework, in which decentralized stochastic policies are trained under a shared centralized value function and executed independently during recovery deployment.
During the training phase, the grid intelligent agent and the gas grid intelligent agent need to share joint state and joint action information to form a unified value judgment for cross network recovery behavior. To this end, a centralized critic is constructed to approximate the joint state–action value function under the current joint policy, serving as a global value baseline for guiding decentralized policy updates. A collaborative recovery evaluation structure is formed by mapping the joint state \(\:{\text{x}}_{\text{t}}\) and the joint action \(\:{\text{A}}_{\text{t}}\), where the joint state \(\:{\text{x}}_{\text{t}}\) is provided by the aforementioned state vector \(\:{\text{S}}_{\text{t}}\). The joint action value function is expressed as \(\:{\text{Q}}_{\theta}\text{(}{\text{x}}_{\text{t}}\text{,}{\text{A}}_{\text{t}}\text{)}\), where \(\:\theta\) denotes the parameters of the centralized critic, and the value estimation is performed based on trajectories generated by the current joint policy. This value function is used to evaluate the long-term impact of different electric gas joint recovery actions on the overall recovery process of the system under the current disaster state.
The centralized critic updates its value estimate through an on-policy temporal difference structure using state–action samples collected under the current joint policy41, and the target value is constructed as:
$$\:{\text{y}}_{\text{t}}\text{=}{\text{r}}_{\text{t}}\text{+}{\gamma}{\text{Q}}_{{\theta}^{\text{-}}}\text{(}{\text{x}}_{\text{t}\text{+1}}\text{,}{\text{A}}_{\text{t}\text{+1}}\text{)}$$
(10)
In Eq. 10, γ is the discount factor, and \(\:{\theta}^{\text{-}}\) is the target network parameter, which is used to alleviate the oscillation problem of value estimation under high disturbance conditions of disasters. This update mechanism enables the central evaluator to gradually form a stable judgment of the quality of collaborative behavior during the recovery process.
To ensure accurate evaluation of the expected return associated with the current joint recovery action, the centralized critic parameters are updated by minimizing the mean square error between the estimated value and the on-policy temporal difference target:
$$\:\text{L}\text{(}\theta\text{)=(}{\text{Q}}_{\theta}\text{(}{\text{x}}_{\text{t}}\text{,}{\text{A}}_{\text{t}}\text{)-}{\text{y}}_{\text{t}}{\text{)}}^{\text{2}}$$
(11)
Equation 11 forms a parameter gradient source to support consistent value judgments of multi-agent systems in a dual network coupled environment.
The electrical system strategy network and the gas system strategy network use their respective local state vectors and local action spaces to execute strategy mappings, with local strategies running in a distributed execution mode. The strategy network outputs an unconstrained action vector \(\:{\text{A}}_{\text{t}}^{\text{u}}\), which is projected through the aforementioned action constraints to form executable actions, ensuring that the local actions of independent agents do not disrupt the coupling energy balance. The policy gradient is computed using advantage estimates derived from the centralized critic, and policy updates are performed by optimizing the expected return under the current stochastic policy. The strategy gradient42 is expressed as \(\:{{\triangledown}}_{{\Phi}}\text{J}\text{=}{{\triangledown}}_{{\Phi}}{\text{Q}}_{\theta}\text{(}{\text{x}}_{\text{t}}\text{,}{\text{A}}_{\text{t}}\text{)}\), where parameter \(\Phi\) is the weight of the strategy network. The gradient is obtained through sampling batch processing, so that the power grid and gas grid strategies converge towards the same optimization direction during continuous training.
After the training phase is completed, the local policy network separates from the central evaluator during the execution phase and relies solely on local information to generate recovery actions. This structural separation directly reflects the regulatory reality of electric–gas coupled systems, in which the power network and the natural gas network are operated by distinct entities with independent dispatch authority, separate control boundaries, and non-shared operational data. During recovery deployment, each subsystem operator executes its own policy using only locally observable state variables without requiring real-time data exchange with the other network, thereby preserving the operational independence and information confidentiality of each regulatory domain. The cross-system coordination learned during centralized training is encoded into the decentralized policy parameters rather than relying on runtime communication, making the execution architecture directly compatible with the jurisdictional boundaries of real-world energy infrastructure management. After the training phase is completed, the local policy network separates from the central evaluator during the execution phase and relies solely on local information to generate recovery actions. This structural separation directly reflects the regulatory reality of electric–gas coupled systems, in which the power network and the natural gas network are operated by distinct entities with independent dispatch authority, separate control boundaries, and non-shared operational data. During recovery deployment, each subsystem operator executes its own policy using only locally observable state variables without requiring real-time data exchange with the other network, thereby preserving the operational independence and information confidentiality of each regulatory domain. The cross-system coordination learned during centralized training is encoded into the decentralized policy parameters rather than relying on runtime communication, making the execution architecture directly compatible with the jurisdictional boundaries of real-world energy infrastructure management. In order to demonstrate the linkage of strategies in the electrical network, this architecture integrates global evaluation information and local decision logic to achieve collaborative optimization of the electric gas system. The corresponding logical structure and information flow are shown in Fig. 3.
Fig. 3
Schematic diagram of centralized training and distributed execution structure.
This architecture consists of a global central evaluator, a power intelligent agent, and a natural gas intelligent agent. The central evaluator obtains joint state and joint action information across networks during centralized training, evaluates the collaboration of multi-agent decisions using a global value function, and guides the evolution of sub agent strategies through a global value guidance mechanism. Electric power intelligent agents and natural gas intelligent agents act as distributed execution units, generating recovery actions based on local observations of their respective networks during the recovery phase. This design achieves global information sharing during the training phase, maintains decision-making independence and real-time performance of each energy system during the execution phase, and alleviates the curse of dimensionality and non steady state problems under strong coupling constraints.
Multi-objective reward construction and curriculum learning and training mechanism
Post disaster recovery decisions not only focus on the rapid recovery of energy supply capacity, but also need to consider the rationality of operational stability and recovery pace. A single indicator is difficult to fully characterize the resilience evolution process of the electric gas system under extreme disasters. Therefore, a multi-objective reward structure is constructed to guide strategies to form coordinated and consistent behavior patterns in different recovery stages.
Multi-objective rewards are constructed from three aspects: energy supply recovery level, operational stability, and temporal action consistency. The goal of energy recovery is reflected through changes in the system’s energy availability, and the incremental functionality during the post disaster recovery process is explicitly incorporated into the return signal. The corresponding recovery increment is defined as43:
$$\:{\text{r}}_{\text{t}}^{\text{e}}\text{=}{\alpha}\text{(}{\eta}_{\text{t}}\text{-}{\eta}_{\text{t}\text{-1}}\text{)}$$
(12)
The parameter \(\:{\eta}_{\text{t}}\) in Eq. 12 is the availability of energy supply, and the parameter α is the weight of recovery increment, which is used to adjust the proportion of recovery speed in the reward.
Pursuing only the speed of energy recovery may lead to frequent violations of safety boundaries at system operating points, thereby introducing steady-state maintenance rewards to suppress overly aggressive recovery behavior. Steady state return characterizes the stability of system operation by the cumulative amount of voltage and pressure offset44:
$$\:{\text{r}}_{\text{t}}^{\text{s}}\text{=-}{\beta}{(\parallel}{\text{v}}_{\text{t}}\text{-}{\text{v}}^{\text{*}}{{\parallel}}_{\text{2}}^{\text{2}}{+\parallel}{\text{p}}_{\text{t}}\text{-}{\text{p}}^{\text{*}}{\parallel}_{\text{2}}^{\text{2}}\text{)}$$
(13)
In Eq. 13, parameters \(\:{\text{v}}^{\text{*}}\) and \(\:{\text{p}}^{\text{*}}\) are the target voltage and pressure, and parameter β is the deviation penalty weight used to maintain operational stability during high disturbance phases.
Disaster recovery has obvious phased characteristics, and sudden changes in recovery actions in the time dimension can easily trigger system oscillations. To constrain the smoothness of the restoration rhythm, a temporal consistency reward is introduced, and the difference in actions between adjacent moments is used as a penalty term45:
$$\:{\text{r}}_{\text{t}}^{\tau}\text{=-}{\delta}\parallel{\text{A}}_{\text{t}}\text{-}{\text{A}}_{\text{t}\text{-1}}{\parallel}_{\text{2}}$$
(14)
The parameter δ in Eq. 14 is the action smoothing weight, used to ensure the controllability of restoring rhythm across time intervals. Three types of returns are superimposed in the joint reward framework to form multi-objective returns:
$$\:{\text{r}}_{\text{t}}\text{=}{\text{r}}_{\text{t}}^{\text{e}}\text{+}{\text{r}}_{\text{t}}^{\text{s}}\text{+}{\text{r}}_{\text{t}}^{{\tau}}$$
(15)
Equation 15 forms a unified feedback signal during the training process of the strategy network, enabling the collaborative evolution of recovery speed, operational stability, and temporal consistency within the same optimization framework.
The impact of post disaster recovery actions often becomes fully apparent after multiple time steps, and relying solely on immediate returns can underestimate the long-term value of critical repair decisions. To characterize the delay effect of recovery behavior, a time diffusion form of reward propagation mechanism is introduced, and the subsequent recovery effect is fed back to the current decision through exponential discount:
$$\:{\text{G}}_{\text{t}}\text{=}\sum\:_{\text{k}\text{=0}}^{\text{K}}\text{}{\lambda}^{\text{k}}{\text{r}}_{\text{t}\text{+}\text{k}}$$
(16)
The parameter λ in Eq. 16 is the delay attenuation coefficient, and the parameter K is the delay propagation step size, which enables the policy network to form a long-term recovery perspective under high disturbance conditions.