\section{Variance lower bound and attainability}
\label{sec:lower_bound}

The examples suggest asking how small the cumulative error covariance
can be. We compare deterministic scalar historical weights applied to
i.i.d. centered innovations $\varepsilon_s\in\mathbb R^p$ with covariance
$Q$, subject to the normalization
\begin{equation}\label{eq:normalized_weights}
b_t=\sum_{s=1}^t w_{t,s}\varepsilon_s,\qquad
\sum_{s=1}^t w_{t,s}=1.
\end{equation}
This normalization preserves constant inputs, as required for a linear
estimator of a common mean. The lower bound below concerns this
specified estimator class.

\begin{proposition}[Variance lower bound for normalized historical weights]
\label{prop:lower_bound}
Under \eqref{eq:normalized_weights}, let
$k_{T,s}=\sum_{t=s}^Tw_{t,s}$ and
$\Gamma_T=\operatorname{Cov}(T^{-1/2}\sum_{t=1}^Tb_t)$.
Then, for every $T\ge1$,
\begin{equation}\label{eq:lower_bound_identity}
\boxed{
\Gamma_T-Q
=\frac QT\sum_{s=1}^T(k_{T,s}-1)^2\succeq0.
}
\end{equation}
In particular,
\begin{equation}\label{eq:attainment_condition}
\frac1T\sum_{s=1}^T(k_{T,s}-1)^2\to0
\quad\Longrightarrow\quad \Gamma_T\to Q.
\end{equation}
If $Q\ne0$, this condition is also necessary for $\Gamma_T\to Q$.
Whenever the downstream PR CLT holds with $C_0=0$ and
$\Gamma_T\to\Gamma$, its covariance satisfies
\begin{equation}\label{eq:downstream_lower_bound}
\Sigma_{\mathrm{PR}}\succeq H^{-1}(S+Q)H^{-\top}.
\end{equation}
\end{proposition}

The running average has cumulative covariance $2Q$, and the fixed-gain
$c/t$ recursion approaches $Q$ only as $c\to\infty$. The next construction
attains $Q$ using a fixed gain schedule, while its single-step MSE decays
more slowly than $t^{-1}$.

\begin{proposition}[Attainment by a polynomial-gain recursion]
\label{prop:attainability}
Let the innovations be as above, fix $0<a\le1$ and $1/2<\rho<1$, and set
\begin{equation}\label{eq:polynomial_gain}
b_1=\varepsilon_1,\qquad
b_t=(1-\gamma_t)b_{t-1}+\gamma_t\varepsilon_t,\quad
\gamma_t=at^{-\rho},\quad t\ge2.
\end{equation}
Equivalently, take $\gamma_1=1$. The resulting deterministic weights are
nonnegative and exactly normalized. They satisfy
\begin{equation}\label{eq:polynomial_attainment}
\mathbb E\|b_t\|^2=O(t^{-\rho}),\qquad
\frac1T\sum_{s=1}^T(k_{T,s}-1)^2=O(T^{\rho-1}),\qquad
\Gamma_T\to Q.
\end{equation}
Thus the lower bound of Proposition~\ref{prop:lower_bound} is attained.
If the objective, SGD step size, and joint innovation conditions of
Theorem~\ref{thm:pr} hold, $C_0=0$, and $\rho>2\alpha-1$, then
\begin{equation}\label{eq:optimal_pr_covariance}
\boxed{
\sqrt T(\bar\theta_T-\theta^*)\Rightarrow
N\!\left(0,H^{-1}(S+Q)H^{-\top}\right).
}
\end{equation}
\end{proposition}

The initialization $b_1=\varepsilon_1$ ensures exact normalization.
The additional condition $\rho>2\alpha-1$ controls the downstream PR
remainders. The three gain schedules compare as follows:
\begin{center}
\small
\begin{tabular}{@{}lll@{}}
\toprule
Gain schedule & Single-step MSE & Cumulative covariance limit \\
\midrule
$1/t$ & $O(t^{-1})$ & $2Q$ \\
$c/t$, $c>1/2$ & $O(t^{-1})$ & $\dfrac{2c}{2c-1}Q$ \\[2mm]
$at^{-\rho}$, $1/2<\rho<1$ & $O(t^{-\rho})$ & $Q$ \\
\bottomrule
\end{tabular}
\end{center}
The third row uses \eqref{eq:polynomial_gain}; the $c/t$ row uses
Example~2 with its cumulatively negligible initialization term.

The bound is sharp within the normalized deterministic historical
linear estimator class.