Robust Bayesian Modeling

What is a Bayesian Model?

• A joint distribution of parameters $\beta$ and data $x$.
• We usually think of exchangable models $$p(\beta,x) = \underbrace{p(\beta|\alpha)}_{\text{prior}}\;\prod_{i=1}^{n} \underbrace{p(x_i|\beta)}_{\text{likelihood}}$$

• Generalise this to accomodate most common models
  • conditional models: $$p(x_i|\beta) \rightarrow p(x_i|\mathbf{y}_i,\beta) \stackrel{\text{e.g.}}{=}\mathcal{N}(x_i|\mathbf{w}^T \mathbf{y}_i,\sigma)$$
  • or latent variable models: $$p(x_i|\beta) = \sum_z p(x_i,z_i|\beta) \stackrel{\text{e.g.}}{=} \sum_k \pi_k \mathcal{N}(x_i|\mu_k,\sigma_k)$$

Interlude: What is the difference between parameters and latent variables?

• Joint distribution: $$p(\beta,\mathbf{z},\mathbf{x}) = \underbrace{p(\beta|\alpha)}_{\text{parameters}} \prod_{i=1}^n \underbrace{p(z_i|\gamma)}_{\text{latent variables}} p(x_i|\beta,z_i)$$
• Practical notion:
  • Number of latent variables grows with number of data points.
  • Number of parameters stays fixed.
• There is no real difference

A useful distinction: Local vs global variables

$$p(\beta,\mathbf{z},\mathbf{x}) = \underbrace{p(\beta|\alpha)}_{\text{global}} \prod_{i=1}^n \underbrace{p(z_i|\gamma)}_{\text{local}} p(x_i|\beta,z_i)$$

• The distinction is determined by conditional dependencies: $$p(x_i,z_i|x_{-i},z_{-i},\beta) = p(x_i,z_i|\beta)$$

Example: Gaussian mixture model

• Which are the local and which are the global variables?

$$\;p(\mathbf{x}|\mathbf{z}) = \prod\limits_k \mathcal{N}(x|\mu_k,\sigma_k)^{z_{k}}; \;\;\;\;\;\; p(\mathbf{z}) = \prod\limits_k \pi_k^{z_k}$$

• global: means $\mu_k$, standard deviations $\sigma_k$, mixture proportions $\pi_k$;
• local: cluster assignments $z_k$.

Motivation

• Wang and Blei, 2015 – A General Method for Robust Bayesian Modeling

• Robustness: Inference should be insensitive to small deviations from the model assumptions.
• Wang and Blei introduce a general framework for robust Bayesian modeling.

Key idea: Localisation of global parameters

• Classical model: $$p(\beta,\mathbf{x}) = p(\beta|\alpha) \prod\limits_{i=1}^n p(x_i|\beta)$$
  • All data is drawn from the parameter.
  • The hyperparmater $\alpha$ is usually fixed.
• Robust model: $$p(\mathbf{\beta},\mathbf{x}) = \prod\limits_{i=1}^n p(\beta_i|\alpha) p(x_i|\beta)$$
  • Every data point is assumed drawn from an individual realisation of the parameter, which is drawn from the prior.
  • Outliers are explained by variation in the parameters.

Graphical Model for Localisation

• Classic model – global $\beta$

• Robust model – local $\beta$

• We now need to fit the hyperparameter $\alpha$.
• Fixing $\alpha$ would make the data points independent.

Example: Normal observation model

• Localise the precision parameter and use the conjugate prior

• Any guesses?

2nd key idea: Empirical Bayes

• Estimate hyperparameters via maximum likelihood $$\hat{\alpha}=\text{arg max}_{\alpha} \sum\limits_{i=1}^{n} \int p(x_i|\beta_i) p(\beta_i|\alpha) d\beta_i$$
• aka evidence approximation $$\text{evidence} = p(x_i|\alpha) = \int p(x_i|\beta_i) p(\beta_i|\alpha) d\beta_i$$
• Here we use the data to determine the prior, is that legit?

Linear Regression

• Trainin data: $\begin{align} y_i|x_i &\sim \mathcal{N}(\omega^T x_i + b, \sigma_i + 0.02) \\ \sigma_i &\sim \text{Gamma}(k,1) \end{align}$
• Test data: $\begin{align} y_i|x_i \sim \mathcal{N}(\omega^T x_i + b, 0.02) \end{align}$

Logistic Regression

$$y_i | x_i \sim \text{Bernoulli}(\sigma(\omega^T x_i))$$