We propose a new concept of codivergence, which quantifies the similarity between two probability measures P₁,P₂ relative to a reference probability measure P₀. In the neighborhood of the reference measure P₀, a codivergence behaves like an inner product between the measures P₁-P₀ and P₂-P₀. Codivergences of covariance-type and correlation-type are introduced and studied with a focus on two specific correlation-type codivergences, the χ²-codivergence and the Hellinger codivergence. We derive explicit expressions for several common parametric families of probability distributions. For a codivergence, we introduce moreover the divergence matrix as an analogue of the Gram matrix. It is shown that the χ²-divergence matrix satisfies a data-processing inequality.

In nonparametric statistics, rate-optimal estimators typically balance bias and stochastic error. The recent work on overparametrization raises the question whether rate-optimal estimators exist that do not obey this trade-off. In this work we consider pointwise estimation in the Gaussian white noise model with regression function f in a class of β-Hölder smooth functions. Let ’worst-case’ refer to the supremum over all functions f in the Hölder class. It is shown that any estimator with worst-case bias ≲n^{−β/(2β+1)}≕ψ_n must necessarily also have a worst-case mean absolute deviation that is lower bounded by ≳ψ_n. To derive the result, we establish abstract inequalities relating the change of expectation for two probability measures to the mean absolute deviation.

It is a common phenomenon that for high-dimensional and nonparametric statistical models, rate-optimal estimators balance squared bias and variance. Although this balancing is widely observed, little is known whether methods exist that could avoid the trade-off between bias and variance. We propose a general strategy to obtain lower bounds on the variance of any estimator with bias smaller than a prespecified bound. This shows to which extent the bias-variance trade-off is unavoidable and allows to quantify the loss of performance for methods that do not obey it. The approach is based on a number of abstract lower bounds for the variance involving the change of expectation with respect to different probability measures as well as information measures such as the Kullback-Leibler or χ²-divergence. Some of these inequalities rely on a new concept of information matrices. In a second part of the article, the abstract lower bounds are applied to several statistical models including the Gaussian white noise model, a boundary estimation problem, the Gaussian sequence model and the high-dimensional linear regression model. For these specific statistical applications, different types of bias-variance trade-offs occur that vary considerably in their strength. For the trade-off between integrated squared bias and integrated variance in the Gaussian white noise model, we propose to combine the general strategy for lower bounds with a reduction technique. This allows us to reduce the original problem to a lower bound on the bias-variance trade-off for estimators with additional symmetry properties in a simpler statistical model. In the Gaussian sequence model, different phase transitions of the bias-variance trade-off occur. Although there is a non-trivial interplay between bias and variance, the rate of the squared bias and the variance do not have to be balanced in order to achieve the minimax estimation rate.