Appendix A:

This appendix provides a brief overview of the fundamental concepts used in this study, including LIME, PPO, TRPO, off-policy PPO, and BOHB. These methods are introduced to support the understanding of the proposed framework and its optimization components.

Appendix A.1: LIME

LIME explains complex model outputs using a simpler surrogate model. This surrogate focuses on the local neighborhood around a particular data point \(x\). Here, \(x\) is an element from the \({X}_{L}\) dataset. The goal of LIME is to reproduce the behavior of the black-box function \(f\). In this study, \(f\) represents the classification model. LIME constructs a locally accurate model \(g\), often linear or interpretable, to approximate \(f\) near \(x\). This step identifies the features that most strongly influence the prediction of \(f\) for the chosen instance.

For a given instance \(x\), LIME generates an interpretation by solving the following optimization problem:

$$\xi (x)=\text{a}\text{r}\text{g}\underset{g\in G}{min} L\left(f,g,{\pi}_{x}\right)+\Omega (g)$$

(26)

In this formulation, \(G\) represents the set of candidate interpretable models. The function \(L\left(f,g,{\pi}_{x}\right)\) measures how closely the surrogate model \(g\) matches the predictions of the original model \(f\). The proximity function \({\pi}_{x}\) assigns weights to perturbed samples depending on their closeness to the instance \(x\). The regularization term \(\Omega (g)\) limits model complexity. This prevents overly complicated explanations and ensures that \(g\) remains interpretable. Here, \(f:{\mathbb{R}}^{d}\to {\mathbb{R}}\) denotes the predictive function being approximated. For a given point \(x\), perturbed versions \(Z=\left\{{z}_{1},{z}_{2},\dots ,{z}_{m}\right\}\) are generated by making small changes to its values. The predictive model \(f\) is applied to each \(z \in Z\) to obtain the corresponding output \(f(z)\). Every perturbed instance receives a weight through the proximity function:

$${\pi}_{x}(z)=\text{e}\text{x}\text{p}\left(-\frac{D(x,z{)}^{2}}{{\sigma }^{2}}\right)$$

(27)

In this setting, \(D(x,z)\) is the distance between the original instance \(x\) and a perturbed sample \(z\). This distance is usually computed using the Euclidean metric. The kernel width parameter \(\sigma \) controls the rate at which weights decay with distance.

The surrogate model \(g\in G\), often linear, is fitted to the weighted dataset \(\{(z,f(z))\hspace{0.17em}\mid \hspace{0.17em}z\in Z\}\). When a linear approximation is employed, \(g\) takes the following form:

$$g(z)={w}_{0}+\sum_{j=1}^{d} {w}_{j}{z}_{j}$$

(28)

In this step, the weights \({w}_{j}\) are updated to minimize the weighted loss function:

$$\widehat{w}=\text{a}\text{r}\text{g}\underset{w}{min} \sum_{z\in Z} {\pi}_{x}(z)(f(z)-g(z;w){)}^{2}+\Omega (g)$$

(29)

The coefficients \({w}_{j}\) show how much each feature contributes to the prediction of \(f\) near \(x\). A positive coefficient indicates that the feature increases the prediction value, whereas a negative coefficient suggests it decreases it. In this way, LIME provides a clear and faithful explanation of the behavior of the model around a chosen instance. By restricting analysis to the neighborhood of \(x\), LIME avoids the difficulty of explaining the global behavior of \(f\). Instead, it highlights the features that most strongly affect the prediction for that instance.

In this framework, LIME is not used as a separate predictive model. Instead, LIME is applied only to generate local importance scores for individual samples, which are then integrated into the reward formulation. The surrogate model \(g\) in LIME is not trained globally. Instead, it is built dynamically around a single instance x using perturbations. Consequently, LIME avoids the need for an additional lengthy training phase. Its function is restricted to producing interpretable coefficients \({w}_{j}\) that reflect the most significant features at a local scale. This design ensures that the values used in the LIME-based reward signal in Eq. 17 stay efficient and reliable, without adding unnecessary computation or complexity.

Figure 

Fig. 18

Fig. 18

Visualization of the LIME process for features \(\{A, B\}\). It shows perturbed samples \(Z\), the proximity-based weighting \({\pi}_{x}\), and the local linear model \(g\) that approximates the black-box model \(f\) around the instance.

18 shows how LIME values are computed for a feature pair \(\{A,B\}\). The horizontal axis represents feature \(A\), and the vertical axis represents feature \(B\), forming a two-dimensional feature plane. Predictions are generated across this space by the black-box model \(f\). LIME creates perturbed samples Z around the target instance \(x\), which is marked as the central point. These samples appear as scattered neighboring points. Each sample \(z\) is weighted using the proximity function \({\pi}_{x}\), with larger weights assigned to points closer to \(x\). A simple surrogate model \(g\), often shown as a linear boundary, is fitted to approximate \(f\) in this local region. The slope of \(g\) indicates the strength of the contribution of features \(A\) and \(B\) to the prediction.

Appendix A.2: PPO, TRPO, and off-policy PPO

In this section, we outline PPO. PPO applies a clipped surrogate objective to control the large policy shifts that can occur in TRPO. Finally, we introduce the off-policy PPO variant. This variant combines features of TRPO and PPO, resulting in more stable and effective training.

Appendix A.2.1 TRPO

In RL, the main goal is to determine a policy \(\pi \) that optimizes the expected sum of discounted rewards75:

$$\eta \left(\pi \right)={E}_{{s}_{0},{a}_{0},\dots }\left[{R}_{0}\right]={E}_{{s}_{0},{a}_{0},\dots }[\sum_{t=0}^{\infty }{\gamma }^{t}r({s}_{t},{a}_{t})]$$

(30)

In this setup, \({s}_{0}\sim {\uprho}_{0}\) represents the initial state sampled from a predefined distribution. The policy \(\pi ({a}_{t}|{s}_{t})\) gives the probability of taking action \({a}_{t}\) in state \({s}_{t}\). The transition function \(P({s}_{t+1}{|s}_{t},{a}_{t})\) describes the probability of reaching the next state given the current state and chosen action. The discount parameter \(\gamma \) regulates the weight of future rewards. TRPO improves the policy by maximizing a surrogate objective. It also controls policy updates with a KL divergence constraint75:

$$\underset{\pi }{\text{max}}{E}_{s\sim {\rho}_{{\pi}_{{\theta}_{old}}}, a \in {\pi}_{{\pi}_{old}}}\left[\frac{{\pi}_{\theta }\left(a|s\right)}{{\pi}_{{\theta}_{old}}\left(a|s\right)}{A}_{{\pi}_{old}}\left(s,a\right)\right]$$

(31)

subject to

$${E}_{s\sim {\rho}_{{\pi}_{old}}}\left[{D}_{KL}({\pi}_{old}(.|s)|| \pi (.|s))\right]\le \delta $$

(32)

In this context, \({\pi}_{old}\) denotes the previous policy, while \(\delta \) specifies the allowable divergence threshold. The term \({D}_{KL}({\pi}_{old}(.|s)|| \pi (.|s))\) represents the KL divergence. It measures how much the new policy \(\pi \) differs from the old policy \({\pi}_{old}\) in state \(s\). The expression \({\rho}_{{\pi}_{old}}\left(s\right)=\sum_{t=0}^{\infty }{\gamma }^{t}P({s}_{t}=s|{s}_{0},{\pi}_{old})\) defines the discounted state distribution. This distribution is obtained when the system starts from the initial state \({s}_{0}\) and follows the old policy. Without this divergence constraint, maximizing the surrogate objective in Eq. 32 could cause large and unstable policy updates.

Appendix A.2.2 PPO

To ensure stable policy adjustments, PPO employs a clipped surrogate objective function75:

$${L}_{PPO}^{CLIP}={E}_{s\sim {\rho}_{{\pi}_{old}, a\sim {\pi}_{old}}}[\text{m}\text{i}\text{n}(\frac{\pi \left(a|s\right)}{{\pi}_{old}\left(a|s\right)}{A}_{{\pi}_{old}}\left(s,a\right),clip(\frac{\pi \left(a|s\right)}{{\pi}_{old}\left(a|s\right)},1-\in ,1+\in ){A}_{{\pi}_{old}}\left(s,a\right))]$$

(33)

The parameter \(\epsilon \) is a small positive constant that helps balance stability and exploration during learning. The advantage function \({A}_{{\pi}_{old}}\left(s,a\right)\) measures how much better action \(a\) in state \(s\) performs compared to the average performance of all actions under the previous policy. Clipping the probability ratio limits large policy deviations and improves training stability75:

$$clip(x, a, b) = max(a, min(b, x))$$

(34)

Here, \(x\) is the variable being constrained. The values \(a\) and \(b\) define the lower and upper bounds of the allowed range. This mechanism discourages policy updates when the probability ratio strays too far from 1. PPO offers several benefits. However, it has a key drawback: a strong dependence on on-policy samples. This dependence increases sample complexity. It forces repeated agent–environment interactions and raises the computational cost of training.

Appendix A.2.3 Off-policy PPO

This section introduces the off-policy PPO algorithm. It aims to improve sample efficiency by using information from past interactions and multiple policy choices. Unlike standard on-policy PPO, which relies mainly on data from ongoing interactions and therefore requires many new samples, off-policy PPO learns from a broader pool of experiences. This pool includes prior trajectories and diverse decision-making strategies. For example, imagine a robot navigating a complicated maze. On-policy PPO updates its behavior only based on the current run, whereas off-policy PPO also incorporates insights from previous attempts made under different conditions. This method accelerates learning by minimizing the need for exhaustive exploration. It also enhances performance by utilizing a broader range of experiences.

Off-policy PPO was selected because it balances sample efficiency, training stability, and suitability for discrete decision-making tasks. Unlike on-policy PPO, it can reuse historical experiences, which is particularly important in AL settings where labeled samples are limited and costly to obtain. Methods such as deep deterministic policy gradient (DDPG) and twin delayed deep deterministic policy gradient (TD3) are primarily designed for continuous control problems and are less suitable for the discrete actions involved in sample selection and feature selection. Although algorithms such as SAC offer strong exploration capabilities, they often require more complex value estimation mechanisms and additional hyperparameter tuning. In contrast, off-policy PPO preserves stable policy updates and improves sample efficiency through experience reuse. These characteristics make it well-suited to the proposed two-agent framework, which requires stable, data-efficient learning for AL and SA.

The off-policy PPO method addresses the optimization problem by maximizing a surrogate objective with off-policy samples. It follows a strategy similar to off-policy TRPO75:

$$\underset{\pi }{\text{max}}{E}_{s\sim {\rho}_{\mu }, a\in \mu }\left[\frac{\pi \left(a|s\right)}{\mu \left(a|s\right)}{A}_{{\pi}_{old}}\left(s,a\right)\right]$$

(35)

Subject to:

$${\overline{D} }_{KL}^{{\rho}_{\mu },sqrt}\left(\mu ,{\pi}_{old}\right){\overline{D} }_{KL}^{{\rho}_{\mu },sqrt}\left({\pi}_{old},\pi \right)+{\overline{D} }_{KL}^{{\rho}_{\mu }}({\pi}_{old},\pi )\le \delta $$

(36)

where

$${\rho}_{\mu }\left(s\right)=\sum_{t=0}^{\infty }{\gamma }^{t}P({s}_{t}=s|{s}_{0},\mu )$$

(37)

$${\overline{D} }_{KL}^{{\rho}_{\mu }}\left({\pi}_{old},\pi \right)={E}_{s\sim {\rho}_{\mu }}[{D}_{KL}\left({\pi}_{old}\left(.|s\right) || \pi \left(.|s\right)\right)]$$

(38)

$${\overline{D} }_{KL}^{{\rho}_{\mu },sqrt}\left(\mu ,{\pi}_{old}\right)={E}_{s\sim {\rho}_{\mu }}[\sqrt{{D}_{KL}\left(\mu \left(.|s\right) || {\pi}_{old}\left(.|s\right)\right)}]$$

(39)

$${\overline{D} }_{KL}^{{\rho}_{\mu },sqrt}\left({\pi}_{old},\pi \right)={E}_{s\sim {\rho}_{\mu }}[\sqrt{{D}_{KL}\left( {\pi}_{old}\left(.|s\right) || \pi \left(.|s\right)\right)}]$$

(40)

Here, \(\mu \) denotes the behavior policy. Without the restriction in Eq. 36, optimizing the surrogate function with off-policy data from Eq. 35 could lead to large and unstable policy shifts. To prevent this, the PPO clipping mechanism is applied. It reshapes the surrogate objective to maintain stability75:

$${L}_{\mu }\left(\pi \right)={E}_{s\sim {\rho}_{\mu }, a\in \mu }[\frac{\pi \left(a|s\right)}{\mu \left(a|s\right)}{A}_{{\pi}_{old}}\left(s,a\right)]$$

(41)

Based on \({L}_{\mu }\left(\pi \right)\) in Eq. 41, the clipped surrogate objective that incorporates off-policy samples can be expressed as 75:

$${\overline{L} }_{\mu }\left(\pi \right)={E}_{s\sim {\rho}_{\mu }, a\in \mu }[\text{m}\text{i}\text{n}(\frac{\pi \left(a|s\right)}{\mu \left(a|s\right)}{A}_{{\pi}_{old}}\left(s,a\right),clip(\frac{\pi \left(a|s\right)}{\mu \left(a|s\right)},1-\in ,1+\in ){A}_{{\pi}_{old}}\left(s,a\right))]$$

(42)

In most cases, the ratio \(\frac{\pi \left(a|s\right)}{\mu \left(a|s\right)}\) falls outside the range \((1-\epsilon ,1+\epsilon )\). As a result, the policy \(\pi \left(a|s\right)\) does not change during optimization of the clipped surrogate function. To solve this problem, Eq. 42 adjusts the clipping bounds \((1-\epsilon ,1+\epsilon )\) by adding a correction factor \(\frac{{\pi}_{{\theta}_{i}}\left(a|s\right)}{\mu \left(a|s\right)}\)75:

$${L}_{Off-Policy PPO}^{CLIP}(\pi )={E}_{s\sim {\rho}_{\mu }, a\in \mu }[\text{m}\text{i}\text{n}[\frac{\pi \left(a|s\right)}{\mu \left(a|s\right)}{A}_{{\pi}_{old}}\left(s,a\right),clip\left(\frac{\pi \left(a|s\right)}{\mu \left(a|s\right)},\frac{{\pi}_{old}\left(a|s\right)}{\mu \left(a|s\right)}(1-\in ),\frac{{\pi}_{old}\left(a|s\right)}{\mu \left(a|s\right)}(1+\in )\right){A}_{{\pi}_{old}}\left(s,a\right)]]$$

(43)

Figure 

Fig. 19

Fig. 19

Off-policy PPO optimization workflow for coordinated enhancement of dual-agent learning in the proposed framework.

19 illustrates a concise, seven-step pipeline for enhancing both agents in the proposed model through the off-policy PPO approach. The workflow starts by defining the optimization target in the model. A surrogate objective is introduced, along with a clipping strategy, to prevent unstable shifts in learning. Next, historical training data are incorporated into the objective, enabling the model to capture broader patterns and generalize more effectively. The ratio of new to old policies is then checked, and adjustments are applied if the ratio exceeds the permitted bounds, thereby promoting controlled, steady updates. This refined policy is continually improved through iterative updates until the system reaches convergence.

Appendix A.3. BOHB

BOHB combines the probabilistic modeling capabilities of BO with the adaptive resource allocation strategy of HB. Conventional BO techniques, such as the tree-structured Parzen estimator (TPE), optimize parameters one at a time. In contrast, BOHB employs a unified kernel density estimator (KDE) to jointly evaluate all parameters. This joint modeling helps the algorithm capture dependencies and interactions between hyperparameters more effectively.

In BOHB, the random sampling step from HB is replaced with a model-guided approach. Here, a KDE proposes promising hyperparameter values by analyzing past evaluations. These candidates are evaluated using the successive halving procedure of HB. In this process, more resources are gradually allocated to the strongest performers across rounds. By combining model-based selection with adaptive allocation, BOHB explores hyperparameter choices more effectively. It balances searching for unexplored configurations (exploration) with focusing on high-potential ones (exploitation). This strategy is particularly effective in complex optimization tasks with limited resources. It finds strong parameter choices with fewer trials than conventional methods.

Algorithm 6 outlines the pseudocode for optimizing hyperparameters with the BOHB method. The goal is to refine a set of parameters \(x\). These may include batch size, learning rate, and other key model settings. A history log \(D\) records tested configurations along with their performance scores. This record begins empty and expands as the algorithm evaluates more candidates. The optimization runs iteratively until a stopping criterion is reached. Examples include a predefined number of iterations or a time limit.

Next, two KDEs are created: \(l(x)\) from \({D}_{good}\) and \(g(x)\) from \({D}_{bad}\). These estimators model the likelihood that a configuration will yield strong or weak performance. The bandwidth of each KDE is modified with a scaling parameter \({b}_{w}\) to expand the search space. The univariate KDE formulation applied in this context is expressed as76:

$$\widehat{f}(x)=\frac{1}{nh}\sum_{i=1}^{n} K\left(\frac{x-{x}_{i}}{h}\right) $$

(44)

In this equation, \(\widehat{f}(x)\) represents the estimated probability density at point x. The variable \(n\) refers to the number of data samples, while \(h\) denotes the bandwidth parameter that determines the smoothness of the density estimate. The kernel function \(K\) is symmetric and non-negative. A common choice is the Gaussian kernel, which integrates to one. Each \({x}_{i}\) corresponds to an observation drawn from the dataset. In BOHB, a multivariate version of KDE is used. This allows the method to evaluate several hyperparameters simultaneously. This version builds a joint probability distribution over the hyperparameter space. It helps reveal dependencies among parameters.

At this stage, the procedure generates \({N}_{s}\) trial hyperparameter configurations using the KDE \(l(x)\). For each candidate \({x}_{candidate}\), the algorithm computes the ratio \(\frac{l\left({x}_{candidate}\right)}{g ({x}_{candidate})}\). This ratio indicates whether the candidate is more likely to perform well or poorly. The candidate with the highest ratio is chosen for evaluation in the following iteration. BOHB repeats this cycle, balancing the search for new parameter options with the refinement of promising ones. This approach enables the rapid identification of effective configurations while keeping computational costs under control.

Algorithm 4

Algorithm 4

Pseudocode for BOHB-based hyperparameter optimization.

Algorithm 6: