This chapter develops a data-driven lower-layer energy management strategy based on an improved TD3 algorithm to realize real-time power allocation for the CAR-EEV. The optimal target speed sequence derived from the upper-layer F-MPC controller is integrated into the agent’s state space to enable proactive decision-making. To address the training instability and overestimation issues inherent in standard TD3 algorithm, the proposed approach incorporates an enhanced priority-based experience replay with multi-buffer classification and an optimal workspace exploration strategy. The resulting algorithm is designed to approximate the globally optimal control performance defined by DP while maintaining real-time responsiveness.
Principle of TD3 algorithm and its adaptation in energy management
Before presenting the proposed enhancements, this section briefly reviews the foundational principles of the standard TD3 algorithm and establishes its adaptation to the CAR-EEV energy management problem. The TD3 algorithm is built upon the actor-critic framework. Within this framework, the actor network maps the current environmental state to the optimal sequence of control actions \(a\). The critic network serves to evaluate the quality of the actions taken by the actor network. The TD3 algorithm, underpinned by its three pillars—a dual-Q network for suppressing overestimation, target policy smoothing to reduce variance, and delayed updates for stable training—provides a robust and stable learning framework for tackling the complex continuous control challenges inherent in CAR-EEV systems.
The algorithm simultaneously learns Q-value through two independent Q-function networks, using the smaller value from the target Q-function network as the update target. \(y\) for the target Q-value used for updating is shown in Eq. (8).
$$y = r_{t} + \gamma \cdot \min \left( {Q_{{\theta_{1}^{\prime } }} (s_{t + 1} ,\tilde{a}^{\prime } ),Q_{{\theta_{2}^{\prime } }} (s_{t + 1} ,\tilde{a}^{\prime } )} \right)$$
(8)
where \({r}_{t}\) is the immediate reward; \(\gamma\) is the discount factor; \({Q}_{{\theta}_{1}{\prime}}(\cdot )\) is the target value network 1 and \({Q}_{{\theta}_{2}{\prime}}(\cdot )\) is the target value network 2; \({s}_{t+1}\) is the next state; \({\widetilde{a}}{\prime}\) is the noisy target action. This “minimization” operation effectively suppresses potential overestimation by individual networks, yielding a more conservative and accurate value target that guides policy updates towards greater reliability.
The second key technology is targeting policy smoothing. To further enhance the stability of value estimation and the robustness of the policy, noise perturbations were incorporated into the noisy target action of TD3. The noisy target action \({\widetilde{a}}{\prime}\) is as follows:
$$\tilde{a}^{\prime } = \mu^{\prime } (s_{t + 1} |\phi^{\prime } ) + \, \varepsilon ,\varepsilon \sim {\text{clip}}({\mathcal{N}}(0,\sigma ), – c,c)$$
(9)
where \({\mu }{\prime}(\cdot )\) is the target policy network; \({\phi }{\prime}\) is a learnable parameter of the target actor network, which determines the target action output by the network based on \({s}_{t+1}\); \(\epsilon\) is noise sampled from a truncated normal distribution; \(\sigma\) is the standard deviation of Gaussian noise and \(c\) is the noise truncation threshold.
The third key technique is delayed policy updates. The TD3 algorithm reduces the update frequency of the actor network to half that of the critic network, effectively preventing policy degradation caused by overestimation bias. The critic network updates both critic networks by minimising the mean squared bellman error loss function \({L}_{i}\).
$$L_{i} = \frac{1}{T}\sum\limits_{k = 1}^{T} {\left( {y – Q_{{\theta_{i} }} (s_{k} ,a_{k} )} \right)^{2} } ,i = 1,2$$
(10)
where \({L}_{i}\) is the loss value of the online critic network; \(T\) is the number of mini-batches; \(y\) is the target Q-value, computed by the target critic network; \({Q}_{{\theta}_{i}}({s}_{k},{a}_{k})\) is the value prediction of the online critic network under state \({s}_{k}\) and action \({a}_{k}\); \({\theta}_{i}\) is the learnable parameter of the corresponding online critic network. By minimising this loss function, the two critic networks can achieve an accurate fit to the value function.
In practical training, the sample mean from the experience replay is used to approximate the expected value under the true state distribution. Therefore, the policy gradient for the actor network can be expressed as:
$$\nabla_{\phi } J(\phi ) \approx \frac{1}{T}\sum\limits_{k = 1}^{T} {\nabla_{a} } Q_{{\theta_{1} }} (s_{k} ,a)|_{{a = \mu_{\phi } (s_{k} )}} \cdot \nabla_{\phi } \mu_{\phi } (s_{k} )$$
(11)
where \({\nabla}_{\phi }J(\phi )\) is the policy gradient of the actor network, which determines the direction of optimisation for the policy network; \(T\) is the number of samples in a mini-batch; \({Q}_{{\theta}_{1}}({s}_{k},a)\) is the gradient of the first online critic network with respect to the action; \({\mu}_{\phi }(\text{s})\) is the deterministic policy output by the online actor network; \({\nabla}_{\phi }{\mu}_{\phi }(\text{s})\) is the gradient of the actor network’s output with respect to the parameter \(\phi\).
To ensure stable network updates, a target network update mechanism is adopted. The soft updates for \({\phi }{\prime}\)、\({\theta}_{1}{\prime}\)、\({\theta}_{2}{\prime}\) (serving as parameters of the target network) are illustrated in Eqs. (12) and (13).
$$\theta_{i}^{\prime } \leftarrow \tau \theta_{i} + (1 – \tau )\theta_{i}^{\prime } ,i = 1,2$$
(12)
$$\phi^{\prime } \leftarrow \tau \phi + (1 – \tau )\phi^{\prime }$$
(13)
where \({\theta}_{i}\) and \({\theta}_{i}{\prime}\) are the trainable parameters of the online critic network and target critic network respectively; \(\phi\) and \({\phi }{\prime}\) represent the parameters of the online actor network and target actor network respectively; τ is the soft update coefficient to control the update rate of target network parameters. The soft update mechanism allows the target network parameters to track the online network parameters smoothly, which effectively improves the stability of algorithm training.
Adapting the TD3 algorithm to the lower layer real-time power allocation problem within CAR-EEV’s hierarchical energy management requires precise definition of its state space, action space, and reward function. The state vector defined in this study is shown in Eq. (14).
$$s_{t} = \left[ {v_{t} ,a_{acc,t} ,T_{{\left\{ {req,t} \right\}}} ,SoC_{t} ,v_{{\{ opt,t:t + H\} }} } \right]$$
(14)
where \({\nu}_{t}\) and \({a}_{acc,t}\) represent the current vehicle speed and acceleration respectively, directly reflecting instantaneous driving demand and inertial load; \({T}_{\left\{req,t\right\}}\) is the required torque calculated based on the current driver pedal input and vehicle speed; \(So{C}_{t}\) is the real-time SoC of the battery, while \({\nu}_{\left\{opt,t:t+H\right\}}\) is the optimal target vehicle speed sequence for the next \(H\) time steps, derived from the upper layer pre-deceleration control algorithm.
To take full advantage of the characteristics of hybrid engines, where the engine speed is decoupled from the wheel speed and the engine can operate along the optimal fuel consumption curve, the engine target power \({P}_{e}\) is selected as the continuous output action of the agent, as shown in Eq. (15).
After the agent selects the engine power point, the corresponding engine speed and torque are obtained by referring to the engine’s optimal fuel consumption curve, ensuring that the engine operates along this curve. With the goal of improving fuel economy and maintaining the SoC, the reward function \({r}_{t}\) is designed as shown in Eq. (16).
$$r_{t} = – (m_{fuel} (t) + \lambda \cdot |P_{batt} (t)| + \alpha \cdot (SoC_{t} – 0.6)^{2} )$$
(16)
where \({m}_{fuel}(t)\) is the engine’s fuel consumption rate; \(\lambda\) is the penalty coefficient for electricity consumption; \(|{P}_{batt}(t)|\) is the absolute value of the battery’s output/input power; \(\alpha\) is the SoC maintenance penalty coefficient, \(So{C}_{t}\) is the real-time SoC and 0.6 is the reference value of the SoC. The setting of coefficients \(\alpha\) and \(\lambda\) determines the relative weighting among the three sub-objectives of fuel economy, battery power stress, and SoC maintenance, requiring meticulous adjustment through experimentation.
Optimal workspace exploration and enhanced priority experience replay
To address the inherent challenges of the standard TD3 algorithm in CAR-EEV energy management tasks, including low sample utilization, imbalance between exploration and exploitation, unstable training process, and vulnerability to local optimum, two synergistic enhancement mechanisms are proposed in this section: optimal workspace exploration (OWE) and enhanced priority experience replay (EPE). The OWE mechanism is designed to delimit a safe and efficient exploration boundary for the agent, eliminating invalid exploration actions from the source to balance exploration and exploitation; the EPE mechanism incorporates multi-buffer classification and adaptive sampling, which explicitly identifies and reinforces learning from high-value experiences while preserving reflection on failed experiences, to accelerate the policy optimization process for TD3 agents in complex energy management tasks.
For the OWE mechanism, the optimal exploration workspace is defined from three dimensions to match the system characteristics of CAR-EEV and the upper-layer F-MPC speed planning results. First, the power demand boundary is determined based on the optimal speed \({v}_{t}\) and acceleration \({a}_{acc,t}\) output by the upper-layer F-MPC controller, which defines the upper and lower limits of engine output power to meet the real-time driving demand:
$$P_{{e,{\text{min}}}} \le P_{e} \le P_{{e,{\text{max}}}}$$
(17)
where \({P}_{e,min}\) is the minimum stable operating power of the engine (5 kW), and \({P}_{e,max}\) is the maximum output power of the engine to meet the current power demand (72 kW), which is updated in real time according to the upper-layer speed planning results.
Second, the engine high-efficiency boundary is set to constrain the engine operating points within the optimal fuel consumption interval of the universal characteristic curve, ensuring the fuel economy of the system. Specifically, this boundary is defined as the region where the engine speed is 2200–2800 rpm and torque is 30–120 N·m. Third, the battery safety boundary is established according to \(So{C}_{t}\), which limits the charge and discharge power of the battery to avoid overcharge (when \(So{C}_{t}\)>0.8) and over-discharge (when \(So{C}_{t}\)<0.3) risks. Based on the defined optimal workspace, the exploration strategy of standard TD3 is improved to constrain the noisy exploration actions strictly within the safe and efficient range, the improved exploration action is formulated as:
$$a_{t} = {\text{clip}}\left( {\mu_{\phi } (s_{t} ) + {\mathcal{N}},P_{{e,{\text{min}}}} ,P_{{e,{\text{max}}}} } \right)$$
(18)
where \({a}_{t}\) is the actual control action executed by the agent; \({\mu}_{\phi }({s}_{t})\) is the deterministic policy action output by the online actor network based on the current state \({s}_{t}\); \(\mathcal{N}\) is the exploration noise, which follows the Gaussian distribution \(\mathcal{N}(0,\sigma )\). \(clip(\cdot )\) is the limiting function that ensures all exploration actions are within the pre-defined optimal workspace. By constraining the exploration range within the optimal workspace, this mechanism theoretically avoids the invalid actions and strategy oscillations of standard TD3 full-space exploration, and improves the training stability and convergence efficiency from the source.
For the EPE mechanism, the experience replay buffer comprises four independent parts: initial buffer, excellent buffer, normal buffer and failure buffer, to classify and reuse the high-quality samples generated by the OWE mechanism efficiently. The exponentially weighted moving average (EWMA) of reward values is used as a criterion to classify experience samples of different levels, which are stored in corresponding experience replay buffers to increase the sampling frequency of high-priority experience samples. Meanwhile, to address the catastrophic forgetting problem, the initial exploration experiences are retained in the initial buffer to enhance the agent’s continuous learning ability. In the EWMA algorithm, the current time step has the largest weighting coefficient, and the weighting coefficient of time steps farther from the current one decreases exponentially with time. It can be used to observe the change trend of variables over time to “smooth” data by reducing noise. The EWMA is shown in Eq. (19).
$$R_{t}^{EWMA} = \beta *R_{t – 1}^{EWMA} + (1 – \beta )*r_{t}$$
(19)
In the equation, \({r}_{t}\) represents the actual value at time \(t\), and the EWMA value at time \(t\) is denoted as \({R}_{t}^{EWMA}\). The coefficient \(\beta\) is the decay weight, where \(\beta\)∈[0,1). Generally, \(\beta\) is initialized to 0.9. If it is set to zero, there is no decay. Substituting the immediate reward value Rt at time \(t\) into Eq. (19), the classification design for experience samples is obtained as shown in Eq. (20).
$$\left\{ {\begin{array}{*{20}l} {R_{std1} = \varphi R_{t}^{EWMA} = \varphi (\beta * R_{t – 1}^{EWMA} + (1 – \beta ) * R_{t} )} \hfill & {} \hfill & {} \hfill \\ {R_{std2} = (1 + \varphi )R_{t}^{EWMA} = (1 + \varphi )(\beta * R_{t – 1}^{EWMA} + (1 – \beta ) * R_{t} )} \hfill & {} \hfill & {} \hfill \\ \end{array} } \right.$$
(20)
\({R}_{t}^{EWMA}\) and \({R}_{t-1}^{EWMA}\) represent the EWMA values of the reward at time \(t\) and \(t-1\) respectively. Based on their magnitudes, \({R}_{std1}\) and \({R}_{std2}\) are designated as the upper and lower classification limits. During the agent’s training, experience samples outside the upper and lower limits are stored in the excellent-experience and failure-experience replay buffers respectively, while the general-experience replay buffer holds samples within this range. The parameter \(\varphi\) for the classification limits lies in the interval (0, 1).
To be clear, the EPE mechanism proposed in this paper is essentially different from the traditional priority experience playback (PER) method: the traditional PER takes TD-error as the priority measure, which is easily affected by noise TD error, and only uses single buffer weighted sampling, which can’t effectively distinguish empirical samples of different stages and different qualities; The EPE mechanism of this study adopts EWMA-based dynamic reward classification as the priority basis, and through the four-buffer hierarchical storage strategy, it not only solves the problem that the traditional PER is susceptible to noise interference and large sampling deviation, but also alleviates catastrophic forgetting by retaining the initial exploration experience. At the same time, it improves the sample utilization rate and controls the growth of computational complexity by using fixed proportion dynamic sampling.
In summary, the proposed OWE and EPE mechanisms form a full-link optimization for the TD3 algorithm: the OWE mechanism ensures the high quality of generated samples from the exploration source, while the priority-focused EPE buffer enhances the agent’s training speed by increasing only the time complexity without altering the space complexity. During algorithm training, the number of samples drawn from the four buffers is dynamically adjusted according to a preset ratio to accelerate algorithm convergence.
Framework for energy optimization EMS based on TD3 algorithm
By integrating the enhanced priority experience buffer with the optimal workspace exploration mechanism, an improved variant of the twin delayed deep deterministic policy gradient algorithm (EMSEPE-TD3) is developed. The overall architecture of the proposed algorithm and its implementation within the EMS for hybrid electric vehicles are illustrated in Fig. 3. In this framework, the ego-vehicle first acquires environmental information through a connected vehicular network, based on which the optimal target velocity profile is generated. Subsequently, the action exploration area planning controller delineates a permissible exploration space for the EPE learning agent. Conditioned on the current environmental state, the agent selects an action \({a}_{t}\) that resides strictly within this constrained exploration region. The vehicle model then executes the action \({a}_{t}\) and returns the corresponding reward signal. The critic network evaluates the value of the action proposed by the actor network according to the predefined reward function. Through iterative action evaluation and parameter adjustment, the agent progressively refines the network parameters to converge toward the optimal control policy.
Fig.3
The framework of EMSEPE-TD3.
The specific parameter design of the EMSEPE-TD3 algorithm is as follows: the reference value of the battery SoC is set to 0.6, the learning rate is set to 0.0001, the update parameter \(\tau\) is 0.005, the discount factor \(\gamma\) is 0.99, and the EWMA parameter \(\beta\) is 0.3. The samples in the experience pool are evenly divided into four parts, with each buffer size being 250,000. The number of samples in the mini-batch set is set to 256. Among them, 62.5% of the samples are selected for the excellent buffer, and 12.5% for each of the remaining three buffers.
The reward function \({r}_{t}\) designed in this chapter can be understood as a single-step surrogate of the global optimal control objective. To clarify this relationship, the energy management problem for CAR-EEV under a given driving cycle can be formulated as a discrete-time finite-horizon optimal control problem. The total journey comprises \(N\) stages, with the state vector \(x(k)=[v(k),a(k),{T}_{req}(k),SoC(k),{v}_{ref}(k)]\) capturing the driving conditions and battery status, and the control variable \(u(k)={P}_{eng}(k)\) representing the engine power command. The global optimization objective is to minimize the total operational cost:
$$J^{*} = \min_{u(0),…,u(N – 1)} \sum\limits_{k = 0}^{N – 1} L \left( {x(k),u(k),w(k)} \right)$$
(21)
where \(w(k)\) is the exogenous demand variables (e.g., upper-layer planned speed profile and traffic conditions). The single-step cost function \(L(\cdot )\) is structurally consistent with the reward function \({r}_{t}\) defined earlier, incorporating both fuel consumption \({m}_{fuel}(k)\) and a battery usage penalty:
$$L(x(k),u(k),w(k)) = m_{fuel} (k) + w_{1} \cdot |P_{batt} (k)| + w_{2} \cdot (SoC(k) – SoC_{ref} )^{2}$$
(22)
where \({w}_{1}\) is the penalty coefficient for electricity consumption and \({w}_{2}\) is the SoC maintenance penalty coefficient.
The DP algorithm can solve this problem to yield the globally optimal policy \({\pi}_{DP}^{*}\). However, DP requires full prior knowledge of the driving cycle and suffers from the curse of dimensionality, making it unsuitable for real-time control. The EMSEPE-TD3 algorithm proposed in this study addresses this limitation by learning a parameterized policy \({\pi}_{\theta }\) that maps the same state space to control actions, trained via the reward function that mirrors the single-step cost. While rigorous optimality guarantees for DRL remain an open theoretical challenge, empirical evidence in Sect. “Verification and results analysis” demonstrates that the learned policy achieves performance close to the DP benchmark in fuel economy, SoC maintenance, and battery life preservation.