Lecture 17: Metropolis Hastings and Regression with Correlated Errors
1 Putting it all together
Goal: estimate the target pdf \[ (U,V)\sim p_0(u,v) \]
1.1 Gibbs vs Metropolis vs Metropolis-Hastings
Gibbs
Given \(\underline{X}^{(s)}=(u^{(s)},v^{(s)})\)
- \(u^{(s+1)}\sim\)
- \(v^{(s+1)}\sim\)
Metropolis
Given \(\underline{X}^{(s)}=(u^{(s)},v^{(s)})\)
- Propose \(u^*\sim\)
- Compute \(r\)
- \(u^{(s+1)}=\begin{cases} & \\ &\\ &\\ & \end{cases}\)
Metropolis-Hastings
2 Regression model with correlated errors
2.1 Historical \(CO_2\) and temperature data
WWaFD
g<-lm(icecore$tmp~icecore$co2)
summary(g)
Call:
lm(formula = icecore$tmp ~ icecore$co2)
Residuals:
Min 1Q Median 3Q Max
-4.2351 -0.9715 0.0082 1.1544 4.4754
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) -23.024140 0.879543 -26.18 <2e-16 ***
icecore$co2 0.079853 0.003834 20.83 <2e-16 ***
---
Signif. codes: 0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
Residual standard error: 1.533 on 198 degrees of freedom
Multiple R-squared: 0.6866, Adjusted R-squared: 0.6851
F-statistic: 433.8 on 1 and 198 DF, p-value: < 2.2e-16Diagnostics
3 Bayesian Analysis (WWaBD)
Prior:
\[ \underline{\beta}\sim MVN(\underline{\beta}_0, \underline\Sigma_0), \quad \sigma^2\sim \text{Inv-Gamma}(\nu_0/2, \nu_0\sigma_0^2/2), \quad \rho\sim \text{Uniform}(0,1) \]
Full conditionals:
\[ \begin{aligned} \left\{\underline{\beta} \mid \mathbf{X}, \underline{y}, \sigma^{2}, \rho\right\} & \sim \text {MVN}\left(\underline{\beta}_{n}, \Sigma_{n}\right), \text { where } \\ \Sigma_{n} & =\left(\Sigma_{0}^{-1}+\mathbf{X}^{T} \mathbf{C}_{\rho}^{-1} \mathbf{X} / \sigma^{2}\right)^{-1} \\ \underline{\beta}_{n} & =\Sigma_{n}\left(\Sigma_{0}^{-1} \underline{\beta}_{0}+\mathbf{X}^{T} \mathbf{C}_{\rho}^{-1} \underline{y} / \sigma^{2}\right), \text { and } \\ \left\{\sigma^{2} \mid \mathbf{X}, \underline{y}, \underline{\beta}, \rho\right\} & \sim \text { Inv-Gamma }\left(\left[\nu_{0}+n\right] / 2,\left[\nu_{0} \sigma_{0}^{2}+\mathrm{SSR}_{\rho}\right] / 2\right), \text { where } \\ \mathrm{SSR}_{\rho} & =(\underline{y}-\mathbf{X} \underline{\beta})^{T} \mathbf{C}_{\rho}^{-1}(\underline{y}-\mathbf{X} \underline{\beta}) . \end{aligned} \]
4 Metropolis + Gibbs
- Update \(\underline{\beta}\) : Sample \(\underline{\beta}^{(s+1)} \sim \operatorname{multivariate} \operatorname{normal}\left(\underline{\beta}_{n}, \Sigma_{n}\right)\), where \(\underline{\beta}_{n}\) and \(\Sigma_{n}\) depend on \(\sigma^{2(s)}\) and \(\rho^{(s)}\).
\[ \\[1cm] \] 2. Update \(\sigma^{2}\) : Sample \(\sigma^{2(s+1)} \sim\) inverse-gamma \(\left(\left[\nu_{0}+n\right] / 2,\left[\nu_{0} \sigma_{0}^{2}+\operatorname{SSR}_{\rho}\right] / 2\right)\), where \(\operatorname{SSR}_{\rho}\) depends on \(\underline{\beta}^{(s+1)}\) and \(\rho^{(s)}\).
\[\\[1cm]\]
- Update \(\rho\) :
Propose \(\rho^{*} \sim\) uniform \(\left(\rho^{(s)}-\delta, \rho^{(s)}+\delta\right)\). If \(\rho^{*}<0\) then reassign it to be \(\left|\rho^{*}\right|\). If \(\rho^{*}>1\) reassign it to be \(2-\rho^{*}\).
Compute the acceptance ratio
\[ r=\frac{p\left(\underline{y} \mid \mathbf{X}, \underline{\beta}^{(s+1)}, \sigma^{2(s+1)}, \rho^{*}\right) p\left(\rho^{*}\right)}{p\left(\underline{y} \mid \mathbf{X}, \underline{\beta}^{(s+1)}, \sigma^{2(s+1)}, \rho^{(s)}\right) p\left(\rho^{(s)}\right)} \]
and sample \(u \sim\) uniform(0,1). If \(u<r\) set \(\rho^{(s+1)}=\rho^{*}\), otherwise set \(\rho^{(s+1)}=\rho^{(s)}\).
5 Fig. 10.9. The first 1,000 values of \(\rho\) generated from the Markov chain.
[1] "Effective Sample size for Beta0, Beta1, Sigma^2, rho:"[1] 52.04805 50.76981 20.17433 23.40762\[ \\[8cm] \]
5.1 Thinned out Markov Chain
5.2 Posterior for \(\beta_2\)
