Part of a series on
Bayesian statistics
Posterior = Likelihood × Prior ÷ Evidence
Background
Bayesian inference Bayesian probability Bayes' theorem Bernstein–von Mises theorem Coherence Cox's theorem Cromwell's rule Principle of indifference Principle of maximum entropy
Model building
Weak prior ... Strong prior Conjugate prior Linear regression Empirical Bayes Hierarchical model
Posterior approximation
Markov chain Monte Carlo Laplace's approximation Integrated nested Laplace approximations Variational inference Approximate Bayesian computation
Estimators
Bayesian estimator Credible interval Maximum a posteriori estimation
Evidence approximation
Evidence lower bound Nested sampling
Model evaluation
Bayes factor Model averaging Posterior predictive
Mathematics portal
v t e

Empirical Bayes methods are procedures for statistical inference in which the prior probability distribution is estimated from the data. This approach stands in contrast to standard Bayesian methods, for which the prior distribution is fixed before any data are observed. Despite this difference in perspective, empirical Bayes may be viewed as an approximation to a fully Bayesian treatment of a hierarchical model wherein the parameters at the highest level of the hierarchy are set to their most likely values, instead of being integrated out.^[1] Empirical Bayes, also known as maximum marginal likelihood,^[2] represents a convenient approach for setting hyperparameters, but has been mostly supplanted by fully Bayesian hierarchical analyses since the 2000s with the increasing availability of well-performing computation techniques. It is still commonly used, however, for variational methods in Deep Learning, such as variational autoencoders, where latent variable spaces are high-dimensional.

Introduction

Empirical Bayes methods can be seen as an approximation to a fully Bayesian treatment of a hierarchical Bayes model.

In, for example, a two-stage hierarchical Bayes model, observed data ${\displaystyle y=\{y_{1},y_{2},\dots ,y_{n}\}}$ are assumed to be generated from an unobserved set of parameters ${\displaystyle \theta =\{\theta _{1},\theta _{2},\dots ,\theta _{n}\}}$ according to a probability distribution ${\displaystyle p(y\mid \theta )\,}$. In turn, the parameters ${\displaystyle \theta }$ can be considered samples drawn from a population characterised by hyperparameters ${\displaystyle \eta \,}$ according to a probability distribution ${\displaystyle p(\theta \mid \eta )\,}$. In the hierarchical Bayes model, though not in the empirical Bayes approximation, the hyperparameters ${\displaystyle \eta \,}$ are considered to be drawn from an unparameterized distribution ${\displaystyle p(\eta )\,}$.

Information about a particular quantity of interest ${\displaystyle \theta _{i}\;}$ therefore comes not only from the properties of those data ${\displaystyle y}$ that directly depend on it, but also from the properties of the population of parameters ${\displaystyle \theta \;}$ as a whole, inferred from the data as a whole, summarised by the hyperparameters ${\displaystyle \eta \;}$.

Using Bayes' theorem,

${\displaystyle p(\theta \mid y)={\frac {p(y\mid \theta )p(\theta )}{p(y)}}={\frac {p(y\mid \theta )}{p(y)}}\int p(\theta \mid \eta )p(\eta )\,d\eta \,.}$

In general, this integral will not be tractable analytically or symbolically and must be evaluated by numerical methods. Stochastic (random) or deterministic approximations may be used. Example stochastic methods are Markov Chain Monte Carlo and Monte Carlo sampling. Deterministic approximations are discussed in quadrature.

Alternatively, the expression can be written as

${\displaystyle p(\theta \mid y)=\int p(\theta \mid \eta ,y)p(\eta \mid y)\;d\eta =\int {\frac {p(y\mid \theta )p(\theta \mid \eta )}{p(y\mid \eta )}}p(\eta \mid y)\;d\eta \,,}$

and the final factor in the integral can in turn be expressed as

${\displaystyle p(\eta \mid y)=\int p(\eta \mid \theta )p(\theta \mid y)\;d\theta .}$

These suggest an iterative scheme, qualitatively similar in structure to a Gibbs sampler, to evolve successively improved approximations to ${\displaystyle p(\theta \mid y)\;}$ and ${\displaystyle p(\eta \mid y)\;}$. First, calculate an initial approximation to ${\displaystyle p(\theta \mid y)\;}$ ignoring the ${\displaystyle \eta }$ dependence completely; then calculate an approximation to ${\displaystyle p(\eta \mid y)\;}$ based upon the initial approximate distribution of ${\displaystyle p(\theta \mid y)\;}$; then use this ${\displaystyle p(\eta \mid y)\;}$ to update the approximation for ${\displaystyle p(\theta \mid y)\;}$; then update ${\displaystyle p(\eta \mid y)\;}$; and so on.

When the true distribution ${\displaystyle p(\eta \mid y)\;}$ is sharply peaked, the integral determining ${\displaystyle p(\theta \mid y)\;}$ may be not much changed by replacing the probability distribution over ${\displaystyle \eta \;}$ with a point estimate ${\displaystyle \eta ^{*}\;}$ representing the distribution's peak (or, alternatively, its mean),

${\displaystyle p(\theta \mid y)\simeq {\frac {p(y\mid \theta )\;p(\theta \mid \eta ^{*})}{p(y\mid \eta ^{*})}}\,.}$

With this approximation, the above iterative scheme becomes the EM algorithm.

The term "Empirical Bayes" can cover a wide variety of methods, but most can be regarded as an early truncation of either the above scheme or something quite like it. Point estimates, rather than the whole distribution, are typically used for the parameter(s) ${\displaystyle \eta \;}$. The estimates for ${\displaystyle \eta ^{*}\;}$ are typically made from the first approximation to ${\displaystyle p(\theta \mid y)\;}$ without subsequent refinement. These estimates for ${\displaystyle \eta ^{*}\;}$ are usually made without considering an appropriate prior distribution for ${\displaystyle \eta }$.

Point estimation

Robbins' method: non-parametric empirical Bayes (NPEB)

Robbins^[3] considered a case of sampling from a mixed distribution, where probability for each ${\displaystyle y_{i}}$ (conditional on ${\displaystyle }$Zdroj:https://en.wikipedia.org?pojem=Empirical_Bayes_method
Text je dostupný za podmienok Creative Commons Attribution/Share-Alike License 3.0 Unported; prípadne za ďalších podmienok. Podrobnejšie informácie nájdete na stránke Podmienky použitia.