Reinforcement learning is increasingly being considered for controlling real-world systems, from fusion plasma
and autonomous vehicles to drug discovery and drinking water treatment, where reliability is essential and tuning
budgets are limited. Actor-critic algorithms share a set of design decisions, such as how the policy is updated,
how it represents the distribution over actions, how its gradient is estimated, and how often it is updated
relative to the value estimator. Using a control task derived from a real water treatment plant, we analyze over
33,000 experiments to determine how these components affect variability across runs and sensitivity to
hyperparameters. Common defaults, such as Gaussian action distributions with pathwise gradient estimators, are
among the least reliable configurations, whereas bounded distributions with adaptive update schedules remain
robust across a wide range of settings. These findings offer empirical guidance to practitioners across
scientific and engineering domains for understanding and making component-level decisions when adapting
actor-critic methods to new real-world control settings.
Problem setting
To isolate the effects of individual components, we study a simplified PID tuning setting. Tuning PID
controllers is a common task in engineered systems, where small changes in parameters can have large effects on
overall stability and efficiency. The environment is constructed from data from a real water treatment system:
the agent outputs PID gains for the backwashing pump controller, a classical PID loop drives the pump, and the
reward summarizes the tracking quality over the rollout. Each rollout begins from a fixed state, reducing the
problem to a continuous-action bandit and stripping away confounding factors such as long-horizon credit
assignment and temporal state dependence.
Schematic of a reinforcement learning system on a water treatment controller.
(A) Deployment setting and agent internals. (B) We vary six components of the learning agent across
33,000+ experiments. (C) Some of the policy parameterizations available to the actor. (D) The
mass-covering vs. mode-seeking KL objectives. Tap to enlarge
Conclusions
We evaluate both the performance and stability of the component choices across hyperparameters. Performance is the mean reward across
seeds (higher is better); stability refers to the run-to-run variability, measured by the coefficient of
variation
across seeds (lower is better). A reliable method should do well on both metrics. We study PPO, SAC,
DDPG, MPO, GreedyAC, vanilla actor-critic, and REINFORCE, with Gaussian, squashed Gaussian,
Student's t, and beta policy parameterizations. Select a conclusion below to read more details.
Citation
@misc{deconstructingac2026,
title={Deconstructing Actor-Critic: A Large-scale Empirical Study of Design Components for Practitioners},
author={Haseeb Shah and Lingwei Zhu and Adam White and Martha White},
year={2026},
eprint={2607.13274},
archivePrefix={arXiv},
primaryClass={cs.LG},
url={https://arxiv.org/abs/2607.13274},
}
Conclusion 1
Mixing immediate reward with value estimates (λ > 0) improves stability and performance.
AC(λ) interpolates between the critic's action-value estimate (λ=0) and the observed reward (λ=1) in the policy gradient. In our experiments, relying entirely on the critic (λ=0) led to high run-to-run variability across all policy parameterizations, with beta policies being particularly affected. This is likely due to value misspecification, introducing bias into the policy update.
Introducing even a moderate reward mixing (λ=0.3) recovers most of the performance gap while significantly reducing variability, with the best effect consistently observed at λ=0.9. Completely removing the critic contribution (λ=1) tends to either reduce performance or increase variability for Gaussian and Student's t policies. See the figure below for the effect of λ on performance and variability and the supplement for a more detailed breakdown across different hyperparameter and environmental noise configurations.
Mixing immediate reward with value estimates improves stability and performance. For each policy parameterization, we fix all the hyperparameters to their default values and vary the value-reward mixing coefficient λ ∈ {0, 0.3, 0.9, 1} in AC(λ). The actor update weights the gradient ∇ ln πθ(At) by λRt + (1−λ) qw(At) − v, interpolating between the critic's action-value estimate (λ=0) and the observed reward (λ=1). Solid lines show mean reward (higher is better, left axis) averaged over 10 seeds while the dashed lines show coefficient of variation (CV = σ/|μ|, lower is better, right axis). The curves are drawn using a shape-preserving cubic interpolation without altering pointwise values. Intermediate values of λ achieve the best combined performance and variability across all policy parameterizations.
Evidence: reward-mixing sweep above · extended AC(λ) breakdown in the supplement · paper
Conclusion 2
Critic inaccuracy is a major contributor to instability.
Inaccurate critics increase run-to-run variability: some runs achieve good performance, whereas others fail, thereby reducing stability. There are several lines of evidence supporting this conclusion. Firstly, increasing λ in AC(λ) reduces the critic's contribution to the policy gradient. The figure in Conclusion 1 shows that higher λ values decrease variability across all policy parameterizations. Secondly, increasing the critic size increases the critic's representational capacity, reducing critic error as long as the critic is trained sufficiently to exploit this added capacity. Because our adaptive critic update trains the critic until its batch error falls below a fixed threshold, the larger network actually realizes this lower error. Consistent with this, the figure below shows that larger critic networks reduce variability across all policy parameterizations in AC(λ).
Finally, note that PPO in our non-contextual bandit setting does not require learning an action-value critic and, correspondingly, its configurations are among the most favorable in the performance-stability trade-off. Three of the four PPO configurations are Pareto efficient in the frontier of Conclusion 3 while the remaining one lies close to the Pareto frontier.
Increasing the critic size reduces run-to-run variability. For each policy parameterization and learning rate, we compare AC(λ) with a small critic (2 hidden layers of 64 units) and a large critic (2 hidden layers of 256 units), with all the hyperparameters fixed at their default values and λ=0. The bar height represents the coefficient of variation (CV) across N=10 seeds, where lower values indicate more stable runs. Larger critics consistently reduce CV across the entire learning-rate range for all policy parameterizations.
Evidence: critic-size sweep above, λ sweep, and the PPO frontier points · paper §Results
Conclusion 3
Beta policies achieve the best performance-stability tradeoff.
We aggregate results over the top-10 hyperparameters for each algorithm, including allowing different UTD per algorithm. In the figure below we observe two patterns. Firstly, the best region of the performance-stability frontier consists entirely of beta configurations, with five different algorithms clustering tightly around near-optimal performance. This region denotes configurations that have high mean reward and low variability across runs and hyperparameters. Secondly, except for AC(λ), no non-beta policy parameterization dominates any beta configuration. The variants of AC(λ) that perform worse are likely due to not sweeping over the actor UTD ratio in that setting.
Some Student's t and Gaussian configurations also lie on the frontier, but only in intermediate regions where they achieve lower mean reward but competitive variability. In contrast, squashed Gaussian, which is the widely-used default in SAC implementations, performs poorly. For a more detailed breakdown, see SAC with the PW estimator, SAC with the LR estimator, PPO, AC(λ) and GreedyAC in the supplement.
Performance-stability frontier across algorithms and policy parameterizations. For each {algorithm, policy} combination, we select the top-10 hyperparameter configurations by mean reward and aggregate over their 100 runs (10 configurations × 10 seeds). The x-axis shows the mean reward (higher is better) averaged across these runs. The y-axis shows the mean coefficient of variation (lower is better), computed by taking the CV across seeds within each configuration and then averaging across 10 configurations (CVmean = (1/N) Σi CVi with N=10). This separates within-configuration variability (run-to-run noise at fixed hyperparameters) from cross-configuration variability (sensitivity to hyperparameter choice). Error bars are 95% bootstrap confidence intervals (n=2000 resamples). Marker shape indicates the algorithm, and color indicates the policy parameterization. The dashed Pareto frontier is computed directly from the plotted aggregate points. The best region of the frontier, with high mean reward and low variability, consists entirely of beta configurations, with five of the six algorithms clustering together. Student's t and Gaussian appear on the frontier only at intermediate performance levels, while squashed policies are strictly dominated by multiple other configurations. Additionally, unlike other algorithms, which are spread apart, PPO configurations are relatively similar in performance across multiple policies.
Evidence: performance-stability frontier above · per-algorithm sweeps in the supplement · paper
Conclusion 4
Bounded-support policy parameterizations are slower to converge compared to unbounded-support ones.
We group the squashed and beta policy parameterizations as bounded-support, and the clipped Gaussian and Student's t as unbounded-support. From the figure below, we observe that the effect depends on how strongly the action-space constraints are enforced by the policy parameterization. With the default actor UTD ratio of 1, the beta policy consistently learns the slowest among all algorithms, followed by squashed policies and then clipped policies. The clipped policies are based on unbounded densities and therefore can be optimized in a relatively unconstrained parameter space. The squashed policies enforce bounds through a smooth non-linear transformation, which introduces additional curvature and sensitivity near the boundaries. In contrast, the beta policy enforces bounds through its density, and its parameters couple together to control the location, shape, and skewness of the distribution simultaneously, making optimization more difficult.
Once we increase the actor UTD ratio (figure below, right), the differences in convergence speed among the various policy parameterizations largely disappear. However, if the critic is not sufficiently accurate, a high actor UTD could amplify the critic's errors and steer the policy towards poor solutions.
Bounded-support policies converge slower than unbounded-support across algorithms. For each {algorithm, policy} combination, we report the number of steps until convergence, defined as the first step at which the return reaches within 90% of the policy's own final 5% average performance. Note that this is a within-policy convergence metric, as the convergence does not guarantee reaching some predefined global threshold. Results are aggregated by pooling all runs from the top-10 hyperparameter configurations for each combination, then computing the mean (height of the lighter bar) and 95% bootstrap confidence intervals (dark bars, n=2000 resamples). Left: at default actor UTD (UTDactor = 1), the policies with bounded support converge slower than unbounded support in most cases, with the beta policy often requiring 2-3x more steps than Gaussian. Right: at UTDactor = 10, this ordering largely collapses and all policies converge at comparable rates. Tap to enlarge
Actor update-to-data ratio (UTD) is a safer tuning knob compared to the actor learning rate.
Scaling the actor learning rate ηactor by k and the actor UTDactor by k are both ways to control how much the parameters are updated. We find scaling either has a very similar effect for the beta policy across the three critic learning rates ηcritic depicted in the figure below.
For a Gaussian policy, however, the behavior is different at high critic learning rates: increasing the actor learning rate produces a significantly stronger performance collapse compared to increasing the actor UTD ratio. This suggests that the Gaussian policy's instability stems from simultaneous fast updates to both the critic and actor networks, rather than from actor parameter movements alone. This indicates that the actor UTD ratio is a safer tuning knob.
Actor update-to-data ratio is a safer tuning knob compared to the actor learning rate. For each policy (Gaussian, beta), we compare two strategies for scaling the actor update magnitude by a factor k ∈ {1, 3, 10, 30}, i.e. scaling the actor learning rate ηactor × k (solid lines) or scaling the actor UTD ratio UTDactor × k (dashed lines), across three representative critic learning rates ηcritic (different panels) and default SAC hyperparameters. Before scaling is applied, the actor and critic learning rates are equivalent, i.e., ηactor = ηcritic. Results are shown as mean reward across 10 seeds, and the shaded bands represent 95% bootstrap confidence intervals with n=1000 resamples. For beta policies, the dashed and solid lines overlap across all learning rates, indicating that the two scaling strategies are almost identical. For Gaussian policies, they are equivalent at low critic learning rates but start diverging at high ηcritic=0.03 (left), where scaling actor LR causes collapse at k ≥ 10 while scaling actor UTD remains relatively stable. This suggests that the Gaussian policy's instability at high actor LR arises due to big updates to both the actor and critic networks at the same time, as opposed to large actor parameter movements alone. Tap to enlarge
Evidence: LR-vs-UTD scaling comparison above · paper §Results
Conclusion 6
Higher actor update-to-data ratio improves beta policies but destabilizes Gaussian policies.
UTDactor has different effects depending on the policy parameterization (figure below). For beta policies, increasing UTDactor from 1 to 30 significantly improves the mean reward and lowers run-to-run variability consistently across PPO, SAC and GreedyAC. In contrast, for Gaussian policies, the mean reward changes minimally while the variability increases significantly, rising by a factor of four for SAC and roughly doubling for GreedyAC and PPO.
Higher actor update-to-data ratio improves beta policies but destabilizes Gaussian policies. We computed results using top-5 learning rates for every {policy, UTDactor} combination while fixing the other hyperparameters to their default values. Left and middle: mean rewards obtained by GreedyAC, SAC and PPO when changing the actor UTD, where higher is better. Each dot is the average over all selected runs. The shaded area represents 95% bootstrap confidence interval over 50 runs with n=2000 resamples. The curves are drawn using a quadratic B-spline interpolation for visual clarity without altering pointwise values. With a Gaussian policy, the impact of changing UTDactor on the average performance is minimal: PPO and GreedyAC decrease, whereas SAC increases. On the other hand, the average performance of the beta policy improves very significantly as UTDactor is increased. Right: mean variability for different actor UTD ratios, where lower is better. Since we are aggregating results over multiple hyperparameter configurations, we compute the variability (CVi = σi/|μi|) separately for each (top-5) hyperparameter configuration and average over these variabilities to give us a single number (CVmean = (1/N) Σi CVi where N=10 configurations). As we increase UTDactor, the variability increases for Gaussian and decreases for beta policy. With the best hyperparameter configurations, the beta policy achieves higher mean reward and lower mean variability than the Gaussian policy across all three learning algorithms. Tap to enlarge
Pathwise gradient estimators are often unreliable with clipped or squashed policies.
Most continuous-control codebases use the pathwise (PW) gradient estimator by default, often without exposing it as a tunable choice. Whether this is a good default, however, depends on the policy parameterization. PW gradients flow through the sampled action, so any transformation that distorts this path such as clipping or squashing can negatively affect the gradient signal. In contrast, the likelihood ratio (LR) gradients are unaffected by this issue, since they do not differentiate through the sampled action.
The figure below shows three regimes. Firstly, clipped policies (Gaussian and Student's t) suffer a significant drop in performance at higher entropies under PW, but remain robust under LR. This is likely because the high entropy pushes more sampled actions to the clipping boundary, where they contribute zero PW gradient. Secondly, squashed policies achieve similar mean rewards under both estimators, but a lower run-to-run variability under the LR estimator. Finally, the beta policy is essentially insensitive to the choice of estimator in both mean performance and variability, likely because its bounds are encoded in the density rather than controlled by a transformation, so there is no clipped or squashed path for the gradient to flow through. The practical implication is to treat PW as a choice rather than a default whenever the policy involves clipping or squashing, and to consider LR in those settings.
Pathwise gradient estimators are often unreliable with clipped and squashed policy parameterizations. We compare pathwise (PW / reparameterized) and likelihood ratio (LR / non-reparameterized) gradient estimators in Soft Actor-Critic by picking the same hyperparameters for each policy parameterization, and display their performance across different entropy values β. Left: absolute difference in mean reward obtained between the two gradient estimators, computed as Δlog10(Reward) = log10(RewardLR) − log10(RewardPW). The curves are drawn using a shape-preserving cubic interpolation to improve visual clarity without altering the pointwise values. The dashed baseline line represents the mean performance of the reparameterization trick variant. The shaded region represents the bootstrap confidence intervals based on n=2000 resamples. A point above the dashed line implies that the corresponding setting has a higher mean performance when used with an LR estimator. We observe that the policies with clipped sampling (Student's t and Gaussian) obtain a higher mean reward with an LR estimator, whereas the policies with natural boundedness (beta) and squashed sampling (squashed Gaussian) behave similarly under both estimators. Middle and right: coefficient of variation (CV = σ/|μ|) of the total reward across training runs for the LR and PW versions, respectively. Bars show the relative variability across all policies at different entropy values, with smaller values indicating a more stable setting. The squashed Gaussian policies are the least stable under both estimators, with a very high variability when used with the PW estimator. Tap to enlarge
Evidence: estimator comparison above · per-policy SAC sweeps in the supplement · paper
Conclusion 8
Adaptive critic updating is more reliable than a fixed critic update-to-data ratio.
Comparing the adaptive critic UTD ratio with the commonly used fixed critic UTD strategy, we observe that adaptive updating largely avoids the failure modes of fixed critic UTD ratios (figure below). The worst case is UTDcritic=1, which produces 11x higher run-to-run variability for Gaussian policies and 5x for beta under UTDactor=10, as well as lower mean reward across configurations. Higher fixed ratios of UTDcritic ∈ {10, 100} perform comparably; however, they require the practitioner to either know or tune the appropriate value in advance. The one case where the fixed ratio can outperform adaptive is beta with UTDactor=1; however, both perform poorly compared to higher UTDactor=10.
Overall, we find adaptive updating replaces the more sensitive UTDcritic hyperparameter with an easier-to-interpret error threshold ω, which can be set based on the magnitude of the critic loss. In principle, a threshold ω defined relative to a normalized reward scale should transfer across environments more naturally than a raw update count, whose appropriate value depends on batch size, critic's representational capacity, and the complexity of the environment.
Adaptive critic updating is more reliable than a fixed critic update-to-data ratio. For each combination of policy, actor UTD ratio UTDactor and critic UTD ratio UTDcritic, we select the best-performing learning rate and aggregate over its 10 seeds. Left: reward gap between the fixed critic UTD ratio UTDcritic and adaptive critic UTD ratio, computed as Δlog10(Reward) = log10(Rewardfixed) − log10(Rewardadaptive), so values above the dashed reference line at zero (adaptive UTD performance) indicate that the fixed ratio outperforms adaptive, and values below indicate the opposite. Solid and dashed curves correspond to UTDactor=1 and UTDactor=10, respectively. Shaded bands represent 95% bootstrap confidence intervals over 50 runs with n=2000 resamples, and curves use shape-preserving cubic interpolation between measured points for visual clarity. Middle and right: coefficient of variation for UTDactor=1 and UTDactor=10, respectively, with adaptive UTD shown as the leftmost pair of bars in each panel for direct comparison. Horizontal dotted lines mark 10% and 30% relative variability as fixed reference points to ease comparison across panels and figures. Fixed critic UTD ratios are competitive with adaptive only for the beta policy at low actor UTD (UTDactor=1). In every other configuration, fixed ratios either yield lower mean reward (left panel, points below baseline) or substantially higher variability (middle and right panels). The Gaussian policy at UTDcritic=1 is the most extreme failure mode, with variability rising to roughly 11× the adaptive value when UTDactor=10. Tap to enlarge
Evidence: fixed-vs-adaptive comparison above · algorithm in paper §Materials & Methods · paper
Conclusion 9 · Supplement
Mirror descent adds complexity without consistent empirical gains.
In the figure below, we observe that increasing the mirror descent period ηmd generally resulted in similar or significantly worse performance. Increasing the mirror descent step size λ (where λ=1/τ) resulted in worse mean rewards at smaller entropies (β ∈ {0.001, 0.01}) but higher at the largest tested entropy (β=0.1). However, the mean reward at the larger entropy is still significantly lower than smaller entropies. Overall, the best performance is achieved at the smallest values of mirror descent stepsize λ and period ηmd. This indicates the stronger the mirror descent, the lower is the resulting performance.
Mirror descent does not consistently improve Soft Actor-Critic. The y-axis shows Δlog10(Reward) = log10(RewardMD) − log10(RewardBaseline), where the rewards on the right-hand side are the rewards averaged over the top three learning rates per hyperparameter configuration. The columns represent changing entropy levels β, the x-axis represents the mirror descent (MD) period ηmd, and the colours represent different MD step sizes λ=1/τ. The curves are drawn using a quadratic B-spline interpolation across MD periods without altering pointwise values. Values above the zero line represent higher average rewards under MD, while the values under represent performance worse than the gradient descent (GD) baseline. At the two lower entropies (β ∈ {0.001, 0.01}), all MD configurations perform worse than GD. The worst case is at 1/τ=0.5 and high ηmd, where the performance drops by roughly 5x compared to GD. At the highest entropy (β=0.1), we observe a small improvement at the lowest MD step sizes (represented by squares), but the absolute performance at this entropy is still significantly lower than the lower entropies. Tap to enlarge
Evidence: mirror-descent sweep above · full entropy breakdown in the supplement
Conclusion 10 · Supplement
Beta policy is more robust to higher learning rates when the critic's contribution is minimized.
PPO and AC(λ > 0.9) show improved robustness to higher learning rates when coupled with the beta policy (see the supplement). In the bandit setting, PPO has no critic, and we observe that the beta policy at lower actor UTD ratios can perform just as well as at higher ratios when the learning rate is tuned. In all the other settings, the beta policy struggled to learn unless the actor UTD ratio was increased.
Evidence: PPO batch-size and AC(λ) sweeps in the supplement
Conclusion 11 · Supplement
Actor UTD reshapes learning-rate sensitivity.
The relationship between learning rate and mean performance is typically similar to an upside-down parabola: with the best setting in the middle, and a poorer performance at the higher and lower extremes. We observe that increasing the actor UTD ratio can alter the shape of this parabola, and in some cases, reduce the sensitivity to learning rate hyperparameter. This implies that in certain settings, an arbitrarily chosen learning rate from a reasonable range is more likely to perform well. We observe this reduction in sensitivity with all the policy losses that we used to analyze UTDactor: SAC with both gradient estimators, PPO, GreedyAC and MPO (see the supplement).
However, the run-to-run variability depends on the policy parameterization. As ηactor is increased, the Gaussian, Student's t, and squashed policies generally exhibit higher variabilities, while the variability of the beta policy is reduced. Due to this, the reduced learning-rate sensitivity is useful only with the beta policy parameterization.
Evidence: per-algorithm learning-rate sweeps (SAC, PPO, GreedyAC, MPO) in the supplement
Conclusion 12 · Supplement
Averaging over more candidate actions stabilizes policy updates.
Across multiple policy losses, we observe that increasing the number of samples used to compute the policy gradient increases the performance and/or reduces overall run-to-run variability. In PPO, increasing the on-policy batch size increases mean performance, particularly at higher entropies. We do not observe an improvement in variability for PPO because it is already too small. In GreedyAC, increasing the number of proposal actions ηtop both improves the performance and reduces variability. Although the improvements are monotonic within our tested ranges, we surmise that there can be diminishing returns or adverse effects if these hyperparameters are increased further.
Evidence: PPO batch-size and GreedyAC proposal-action sweeps in the supplement
