\section{Related work}
\label{sec:related_work}

Ruppert~\cite{ruppert1988} and Polyak and Juditsky~\cite{polyak1992}
established the classical averaging
principle and its asymptotic efficiency under stochastic approximation
conditions. Kushner and Yang~\cite{kushner1993} extend averaging results
to more general stochastic approximation processes. Mou et al.~\cite{mou2020} give a refined covariance analysis
for averaged linear stochastic approximation with a fixed step size.
These results provide the background for understanding how averaging
transforms stochastic updates into a limiting covariance.

Coupled stochastic approximation is particularly relevant when one
recursion supplies estimates to another. Konda and Tsitsiklis~\cite{konda2004}
establish asymptotic covariance and normality results for linear
two-time-scale recursions. Mokkadem and Pelletier~\cite{mokkadem2006}
extend the analysis to nonlinear recursions; their Theorem~2 establishes
a joint CLT for the two averaged components. Their later
work~\cite{mokkadem2011} uses two-time-scale algorithms to construct
asymptotically efficient estimators. This theory provides a
closely related treatment of recursive auxiliary estimation and its
propagation into an averaged downstream process.

Biased-gradient analyses, such as Ajalloeian and Stich~\cite{ajalloeian2020},
quantify the effects of gradient errors on convergence and attainable
accuracy. Online auxiliary estimation also appears in stochastic
compositional gradient descent, where Wang, Fang, and Liu~\cite{wang2017}
track an inner expectation recursively, and in nuisance-dependent
optimization, where Yu et al.~\cite{yu2025} study convergence and gradient
orthogonalization. These settings motivate the study of estimation
errors that persist across successive gradient evaluations.

The focus of this note is the temporal correlation structure generated
by deterministic historical weights. Cumulative weights give a direct
expression for the PR covariance, including cross-covariance with the
SGD noise, and make the effect of auxiliary-estimator gains explicit.

\subsection{Our Contributions}
\label{sec:contributions}

\begin{enumerate}
\item \textbf{Historical correlation and preservation of SGD guarantees.}
We study vanishing gradient errors generated by deterministic weighted
histories of random innovations. We give moment and cumulative-weight
conditions that preserve mean-square and almost-sure convergence and
$\sqrt T$-asymptotic normality of the PR average, and characterize the
resulting covariance. The classical PR covariance is recovered when the
normalized cumulative error vanishes, in particular when
$\mathbb E\|b_t\|^2=O(t^{-\beta})$ with $\beta>1$ under the stated
regularity conditions.
\item \textbf{A sharp downstream variance bound and an attaining estimator.}
Within the normalized deterministic historical linear estimator class,
we establish a cumulative variance lower bound and construct a recursive
estimator attaining it. Under zero cross-covariance with the SGD noise
and the PR conditions, this construction minimizes the downstream
asymptotic covariance. Counterintuitively, its single-step estimation
MSE decays at order $t^{-\rho}$, $1/2<\rho<1$, while a running mean
achieves $t^{-1}$ but has a larger downstream covariance. Thus the
estimator with the faster single-step MSE rate can yield the less
efficient PR average.
\end{enumerate}