\begin{abstract}
Online estimation can improve the accuracy of individual gradient estimates
while increasing the asymptotic variance of averaged stochastic gradient
descent (SGD). We study this phenomenon through a weighted historical model
for vanishing correlated gradient errors. Under strong convexity and
smoothness, we establish mean-square and almost-sure convergence and derive
a Polyak--Ruppert central limit theorem with an explicit covariance
determined by cumulative historical weights. Running averages and recursive
weighted averages illustrate how estimator gains affect single-step
accuracy and downstream efficiency differently. We also establish a sharp
variance lower bound within the class of normalized deterministic historical
linear estimators and construct a recursion that attains it. Counterintuitively,
this recursion has a slower single-step estimation MSE rate than the running
mean, yet achieves the minimum downstream PR asymptotic variance in that
class. Experiments
on a nonlinear stochastic optimization problem confirm the predicted
covariance formulas and demonstrate that minimizing single-step estimation
error need not minimize the asymptotic variance of the averaged SGD iterate.
\end{abstract}