Complex-Corrected Stochastic Gradient Descent for Noisy Observations

# Complex-corrected stochastic gradient descent ## Problem, assumptions and results Minimize $F(\theta)=\mathbb E\ell(\theta;Z)$, $\theta\in\mathbb R^p$, observing $W=Z+U$ with $U\sim N(0,\Omega)$ independent of sub-Gaussian $Z\in\mathbb R^d$. The known matrix $\Omega\succeq0$ can be singular and nondiagonal. Write $g=\nabla_\theta\ell$. Section 3 assumes an entire input extension $g^{\mathbb C}$, real on real inputs, and fixed finite constants $a,A,B$ satisfying $$ \|D_qg^{\mathbb C}(\theta;q)\|_{\rm op}\le M(\theta)e^{a\|q\|},\qquad M(\theta)^2\le A+B\|\nabla F(\theta)\|^2. $$ The exponent $a$ is independent of $\theta$. Differentiation and expectation commute, and the clean centered score satisfies $$ \mathbb E\|g(\theta;Z)-\nabla F(\theta)\|^2\le\sigma_0^2+\rho_0\|\nabla F(\theta)\|^2. $$ The algorithm samples fresh mutually independent data and $V_t\sim N(0,\Omega)$ at every step, independently of the past $\mathcal F_{t-1}$: $$ G_t=\operatorname{Re}g^{\mathbb C}(\theta_t;W_t+iV_t),\qquad \theta_{t+1}=\theta_t-\eta_tG_t. $$ The Gaussian corrected-score identity is classical; the manuscript's contribution is the executable oracle together with explicit moment estimates and stochastic-approximation analysis. No claim of historical priority is added here. For deterministic $\theta_1$ and $\eta_t=c/(t+\gamma)^\alpha$, $c>0$, $\gamma\ge0$, $1/2<\alpha\le1$, Theorems 4.1–4.5 give: - Convex $L$-smooth $F$ with a minimizer: for $\eta_1\le[L(1+\rho)]^{-1}$ and the step-weighted average $\bar\theta_T^\eta$, $$ \mathbb E F(\bar\theta_T^\eta)-F_*\le \frac{\|\theta_1-\theta^*\|^2+\sigma^2\sum_{t\le T}\eta_t^2}{2\sum_{t\le T}\eta_t}. $$ - Under $\mu$-strong convexity and $L$-smoothness, $\theta_t\to\theta^*$ almost surely and in mean square. MSE is $O(t^{-\alpha})$ for $\alpha<1$; at $\alpha=1$ it is $O(t^{-1})$, $O(t^{-1}\log t)$, or $O(t^{-2\mu c})$ according as $2\mu c>1$, $=1$, or $<1$. - For $L$-smooth $F\ge F_{\inf}$ and the same small-step restriction, $$ \frac{\sum_{t\le T}\eta_t\mathbb E\|\nabla F(\theta_t)\|^2}{\sum_{t\le T}\eta_t} \le\frac{2(F(\theta_1)-F_{\inf})+L\sigma^2\sum_{t\le T}\eta_t^2}{\sum_{t\le T}\eta_t}. $$ The convex and stationarity rates are $O(T^{\alpha-1})$ for $\alpha<1$, and $O(1/\log T)$ at $\alpha=1$. Stationarity does not assert convergence to a global minimizer. The two CLTs additionally require a locally Lipschitz Hessian near $\theta^*$ and continuity there of the corrected-noise covariance $S(\theta)$. Put $H=\nabla^2F(\theta^*)$ and $S_*=S(\theta^*)$. Last-iterate scaling is $t^{\alpha/2}$ and covariance solves $$ H\Sigma_c+\Sigma_cH=cS_*\quad(\alpha<1). $$ At $\alpha=1$, require $c\lambda_{\min}(H)>1/2$; scaling is $\sqrt t$ and $$ (cH-\tfrac12I)\Sigma_c+\Sigma_c(cH-\tfrac12I)=c^2S_*. $$ For the uniform average $\bar\theta_T=T^{-1}\sum_{t\le T}\theta_t$, the $\sqrt T$ covariance is $H^{-1}S_*H^{-1}$ for $\alpha<1$, and the kernel covariance below for $\alpha=1$ under the same spectral condition. ## Core mechanisms and essential derivations ### 1. Rotational invariance gives exact correction (Lemma 3.4; Appendix A.1) Let $D=U+iV$. The joint Gaussian covariance of $(U,V)$ is $\operatorname{diag}(\Omega,\Omega)$, hence $e^{i\phi}D\overset d=D$, even for singular $\Omega$. For each fixed $z,d$, the scalar-direction restriction $s\mapsto g^{\mathbb C}(\theta;z+sd)$ is entire, so $$ \frac1{2\pi}\int_0^{2\pi}g^{\mathbb C}(\theta;z+e^{i\phi}d)\,d\phi=g(\theta;z). $$ The derivative envelope yields $$ \|g^{\mathbb C}(\theta;q)-g(\theta;0)\|\le M(\theta)\|q\|e^{a\|q\|}, $$ which is integrable under the Gaussian perturbation. Fubini and rotational invariance give exact conditional centering, first given $Z_t$, then given the past. Thus $\mathbb E[G_t\mid\mathcal F_{t-1}]=\nabla F(\theta_t)$. No convexity is needed for this identity. The reusable pattern is symmetry plus a holomorphic mean value with explicit domination. ### 2. Conditioning removes the variance cross term (Section 4; Appendix A.2) Decompose $\xi_t=G_t-\nabla F(\theta_t)=\Delta_t+\zeta_t$, where $\Delta_t=G_t-g(\theta_t;Z_t)$ and $\zeta_t=g(\theta_t;Z_t)-\nabla F(\theta_t)$. Since $\mathbb E[\Delta_t\mid\mathcal F_{t-1},Z_t]=0$, the two components are conditionally orthogonal. The fundamental theorem of calculus gives $$ \|\Delta_t\|\le M(\theta_t)\|D_t\|e^{a(\|Z_t\|+\|D_t\|)},\quad K_a=\mathbb E e^{2a\|Z\|}\mathbb E[\|D\|^2e^{2a\|D\|}]<\infty. $$ Consequently $$ \mathbb E_t\|\xi_t\|^2\le\sigma^2+\rho\|\nabla F(\theta_t)\|^2,\qquad \sigma^2=\sigma_0^2+K_aA,\quad\rho=\rho_0+K_aB. $$ Pointwise moment finiteness is insufficient for uniform deterministic constants along a trajectory; the amplitude control supplies them. For higher moments, subtract $g(\theta;0)$ before bounding the centered score. With $b_a(q)=\|q\|e^{a\|q\|}$ and $Q=Z+U+iV$, $$ \|\xi(\theta)\|\le M(\theta)[b_a(Q)+\mathbb E b_a(Z)],\qquad \mathbb E_t\|\xi_t\|^r\le C_r[A+B\|\nabla F(\theta_t)\|^2]^{r/2},\quad r\ge2. $$ Strong convergence is proved first by the squared-distance recursion. It then supplies a finite random trajectory bound for these moments; the CLT does not assume stability circularly. For $n$ independent observations and $m$ independent imaginary simulations per observation, the law of total variance gives $$ \operatorname{Var}_{\rm trace}(G^{(n,m)})=v_{\rm obs}/n+v_{\rm sim}/(nm). $$ Repeating $V$ at one observed $W$ reduces only $v_{\rm sim}$, not the observation uncertainty. ### 3. Localization preserves the correct endpoint hypothesis (Appendices A.5–A.7) At $\alpha=1$, the local spectral condition permits $m\in(1/(2c),\lambda_{\min}(H))$. On a sufficiently small ball, the stopped squared-error recursion yields $O_N(t^{-1})$ second moments and nonlinear remainders, while the probability of leaving the ball after time $N$ tends to zero. This avoids imposing the stronger global requirement $2\mu c>1$ in the CLT. The uniform average retains the full accumulated product weights. Their limit is $$ K_c(u)=c\,u^{cH-I}\int_u^1v^{-cH}\,dv,\qquad \Xi_c=\int_0^1K_c(u)S_*K_c(u)^\top\,du. $$ A bound $\|K_c(u)\|\le C u^{\nu-1}$ for $1/2<\nu<\min\{c\lambda_{\min}(H),1\}$ controls the small-$u$ tail in $L^2$. Compact convergence alone would not justify the covariance limit. In the scalar specialization $cH=1$, $K_c(u)=-c\log u$ and $\Xi_c=2c^2S_*$, so the interior PR formula cannot be reused at the endpoint. ### 4. Reuse boundaries (Section 7.1; Appendix A.9) An observed response $Y$ is held fixed during input complexification. Conditional centering must condition on $(Z,Y)$; joint moments require separate verification. The affine-parameter regression example verifies them directly. For non-Gaussian scalar noise, use an admissible independent conjugate with characteristic function $1/M_U(t)$. Polynomial cancellation extends to entire scores only with justified Taylor/expectation interchange. Uniform noise on $[-b,b]$ has conjugate exponential-moment threshold $\lambda=\pi/b$; symmetric $\pm b$ noise has $\lambda=\pi/(2b)$. The sufficient envelope conditions are $a<\lambda$ for the identity, $2a<\lambda$ for variance, and $(2+\delta)a<\lambda$ for a CLT moment. These are sufficient conditions, not converse divergence claims. Use the analytic score in the noisy input, take its real part after evaluation, and do not replace an analytic square by a modulus square. Trainable sine networks need not satisfy a parameter-independent envelope; their experiments do not inherit the theorem automatically. Experiments were not rerun for this upload.