Statistics/Point Estimation

Introduction

Usually, a random variable $X$ resulting from a random experiment is Template:Colored em to follow a certain distribution with an unknown (but Template:Colored em) parameter (vector) $\theta \in {ℝ}^k$ ($k$ is a positive integer, and its value depends on the distribution), taking value in a set $\mathrm{\Theta }$, called the parameter space. Template:Colored remark For example, suppose the random variable $X$ is assumed to follow a normal distribution $𝒩(\mu ,{\sigma }^2)$. Then, in this case, the parameter vector $\theta =(\mu ,\sigma )\in \mathrm{\Theta }$ is unknown, and the parameter space $\mathrm{\Theta }=\{(\mu ,\sigma ):\mu \in ℝ,\sigma >0\}$. It is often useful to Template:Colored em those unknown parameters in some ways to "understand" the random variable $X$ better. We would like to make sure the estimation should be "good" enough, so that the understanding is more accurate.

Intuitively, the (realization of) Template:Colored em $X_1,\dots ,X_n$ should be useful. Indeed, the estimators introduced in this chapter are all based on the random sample in some sense, and this is what Template:Colored em mean. To be more precise, let us define Template:Colored em and Template:Colored em. Template:Colored definition Template:Colored remark Template:Colored example In the following, we will introduce two well-known point estimators, which are actually quite "good", namely Template:Colored em and Template:Colored em.

Maximum likelihood estimator (MLE)

As suggested by the name of this estimator, it is the estimator that Template:Colored em some kind of "likelihood". Now, we would like to know what "likelihood" should we maximize to estimate the unknown parameter(s) (in a "good" way). Also, as mentioned in the introduction section, the estimator is based on the random sample in some sense. Hence, this "likelihood" should be also based on the random sample in some sense.

To motivate the definition of maximum likelihood estimator, consider the following example. Template:Colored example Template:Colored remark Intuitively, with these particular realizations (fixed), we would like to find a value of $p$ that maximizes this probability, i.e.,, makes the realizations obtained to be the one that is "most probable" or "with maximum likelihood". Now, let us formally define the terms related to MLE. Template:Colored definition Template:Colored remark Template:Colored definition Template:Colored remark Now, let us find the MLE of the unknown parameter $p$ in the previous coin flipping example. Template:Colored example Sometimes, there is constraint imposed on the parameter when we are finding its MLE. The MLE of the parameter in this case is called a Template:Colored em MLE. We will illustrate this in the following example. Template:Colored example To find the MLE, we sometimes use methods other than derivative test, and we do not need to find the log-likelihood function. Let us illustrate this in the following example. Template:Colored example In the following example, we will find the MLE of a parameter vector. Template:Colored example Template:Colored exercise Template:Colored example

Method of moments estimator (MME)

For maximum likelihood estimation, we need to utilize the likelihood function, which is found from the joint pmf of pdf of the random sample from a distribution. However, we may not know exactly the pmf of pdf of the distribution in practice. Instead, we may just know some information about the distribution, e.g. mean, variance, and some moments ($r$th moment of a random variable $X$ is $𝔼[X^r]$, we denote it by ${\mu }_r$ for simplicity). Such moments often contain information about the unknown parameter. For example, for a normal distribution $𝒩(\mu ,{\sigma }^2)$, we know that $\mu ={\mu }_1$ and ${\sigma }^2={\mu }_2-({\mu }_1{)}^2$. Because of this, when we want to estimate the parameters, we can do this through estimating the moments.

Now, we would like to know how to estimate the moments. We let $m_r=\frac{\sum _{i=1}^n{X}_i^r}{n}$ be the $r$th Template:Colored em , where $X_i$'s are independent and identically distributed. By Template:Colored em (assuming the conditions are satisified), we have

• $\bar{X}=m_1\stackrel{p}{\to }𝔼[X]={\mu }_1$
• $m_2\stackrel{p}{\to }𝔼[X^2]={\mu }_2$ (this can be seen from replacing the "$X$" by "$X^2$" in the weak law of large number, then the conditions are still satisfied, and so we can still apply the weak law of large number)

In general, we have $m_r\stackrel{p}{\to }{\mu }_r$, since the conditions are still satisfied after replacing the "$X$" by "$X^r$" in the weak law of large number.

Because of these results, we can estimate the $r$-th moment ${\mu }_r$ using the $r$-th sample moment $m_r$, and the estimation is "better" when $n$ is large. For example, in the above normal distribution example, we can estimate $\mu$ by $m_1$ and ${\sigma }^2$ by $m_2-(m_1{)}^2$, and these estimators are actually called the Template:Colored em.

To be more precise, we have the following the definition of the Template:Colored em: Template:Colored definition Template:Colored remark Template:Colored example Template:Colored remark Template:Colored example Template:Colored exercise

Properties of estimator

In this section, we will introduce some criteria for evaluating how "good" a point estimator is, namely Template:Colored em, Template:Colored em and Template:Colored em.

Unbiasedness

For $\hat{\theta }$ to be a "good" estimator of a parameter $\theta$, a desirable property of $\hat{\theta }$ is that its expected value equals the value of the parameter $\theta$, or at least close to the value. Because of this, we introduce a value, namely Template:Colored em, to measure how close is the mean of $\hat{\theta }$ to $\theta$. Template:Colored definition We will also define some terms related to bias. Template:Colored definition Template:Colored definition Template:Colored remark Template:Colored example Template:Colored example

Efficiency

We have discussed how to evaluate the unbiasedness of estimators. Now, if we are given two unbiased estimators, $\hat{\theta }$ and $\tilde{\theta }$, how should we compare their goodness? Their goodness is the same if we are only comparing them in terms of unbiasedness. Therefore, we need another criterion in this case. One possible way is to compare their Template:Colored em, and the one with smaller variance is better, since on average, the estimator is less deviated from its mean, which is the value of the unknown parameter by the definition of unbiased estimator, and thus the one with smaller variance is more accurate in some deviation sense. Indeed, an unbiased estimator can still have a large variance, and thus deviate a lot from its mean. Such estimator is unbiased since the positive deviations and negative deviations somehow cancel out each other. This is the idea of Template:Colored em.

Template:Colored definition Template:Colored remark Actually, for the variance of unbiased estimator, since the mean of the unbiased estimator is the unknown paramter $\theta$, it measures the mean of the squared deviation from $\theta$, and we have a specific term for this deviation, namely Template:Colored em (MSE). Template:Colored definition Template:Colored remark Notice that in the definition of MSE, we do not specify that $\hat{\theta }$ to be an unbiased estimator. Thus, $\hat{\theta }$ in the definition may be biased. We have mentioned that when $\hat{\theta }$ is unbiased, then its variance is actually its MSE. In the following, we will give a more general relationship between $\mathrm{MSE}(\hat{\theta })$ and $\mathrm{Var}(\hat{\theta })$, not just for unbiased estimators. Template:Colored proposition

Proof. By definition, we have $\mathrm{MSE}(\hat{\theta })=𝔼[(\hat{\theta }-\theta {)}^2]$ and $\mathrm{Var}(\hat{\theta })=𝔼[(\hat{\theta }-𝔼[\hat{\theta }]{)}^2]$. From these, we are motivated to write $$\begin{array}{cc}\mathrm{MSE}(\hat{\theta })& =𝔼[(\hat{\theta }-\theta {)}^2]\\ & =𝔼\left[\right((\hat{\theta }-𝔼[\hat{\theta }])+(𝔼[\hat{\theta }]-\theta ){)}^2]\\ & =𝔼[(\hat{\theta }-𝔼[\hat{\theta }]{)}^2+2(\hat{\theta }-𝔼[\hat{\theta }])\underset{\text{constant}}{\underbrace{(𝔼[\hat{\theta }]-\theta )}}+(𝔼[\hat{\theta }]-\theta {)}^2]\\ & =\mathrm{Var}(\hat{\theta })+2(𝔼[\hat{\theta }]-\theta )\underset{=𝔼[\hat{\theta }]-𝔼[\hat{\theta }]=0}{\underbrace{𝔼[\hat{\theta }-𝔼[\hat{\theta }]]}}+[\mathrm{Bias}(\hat{\theta }){]}^2\\ & =\mathrm{Var}(\hat{\theta })+[\mathrm{Bias}(\hat{\theta }){]}^2,\end{array}$$ as desired.

$◻$

Proof.

• "if" part is simple. Assume $\underset{n\to \mathrm{\infty }}{\lim }\mathrm{Var}(\hat{\theta })=0$ and $\underset{n\to \mathrm{\infty }}{\lim }\mathrm{Bias}(\hat{\theta })=0$. Then, $\underset{n\to \mathrm{\infty }}{\lim }(\mathrm{Var}(\hat{\theta })+(\mathrm{Bias}(\hat{\theta }){)}^2)=0\Rightarrow \underset{n\to \mathrm{\infty }}{\lim }\mathrm{MSE}(\hat{\theta })=0$.
• "only if" part: we can use proof by contrapositive, i.e., proving that if $\underset{n\to \mathrm{\infty }}{\lim }\mathrm{Var}(\hat{\theta })\ne 0$ Template:Colored em $\underset{n\to \mathrm{\infty }}{\lim }\mathrm{Bias}(\hat{\theta })=0$, then $\underset{n\to \mathrm{\infty }}{\lim }\mathrm{MSE}(\hat{\theta })\ne 0$.

• Case 1: when $\underset{n\to \mathrm{\infty }}{\lim }\mathrm{Var}(\hat{\theta })\ne 0$, it means $\underset{n\to \mathrm{\infty }}{\lim }\mathrm{Var}(\hat{\theta })>0$ since the variance is nonnegative. Also, $\underset{n\to \mathrm{\infty }}{\lim }(\mathrm{Bias}(\hat{\theta }){)}^2\ge 0$. It follows that $\underset{n\to \mathrm{\infty }}{\lim }\mathrm{MSE}(\hat{\theta })>0$, i.e., the MSE does not equal zero.
• Case 2: when $\underset{n\to \mathrm{\infty }}{\lim }\mathrm{Bias}(\hat{\theta })\ne 0$, it means $\underset{n\to \mathrm{\infty }}{\lim }(\mathrm{Bias}(\hat{\theta }){)}^2>0$. Also, $\underset{n\to \mathrm{\infty }}{\lim }\mathrm{Var}(\hat{\theta })\ge 0$. It follows that $\underset{n\to \mathrm{\infty }}{\lim }\mathrm{MSE}(\hat{\theta })>0$, i.e., the MSE does not equal zero.

$◻$

Uniformly minimum-variance unbiased estimator

Now, we know that the smaller the variance of an unbiased estimator, the more efficient (and "better") it is. Thus, it is natural that we want to know what is the Template:Colored em efficient (i.e., the "best") unbiased estimator, i.e., the unbiased estimator with the smallest variance. We have a specific name for such unbiased estimator, namely Template:Colored em . To be more precise, we have the following definition for UMVUE: Template:Colored definition Indeed, UMVUE is Template:Colored em, i.e., there is exactly one unbiased estimator with the smallest variance among all unbiased estimators, and we will prove it in the following. Template:Colored proposition

Proof. Assume that $W$ is an UMVUE of $\tau (\theta )$, and $W^{\prime }$ is another UMVUE of $\tau (\theta )$. Define the estimator $W^{*}=\frac{1}{2}(W+W^{\prime })$. Since $𝔼[W^{*}]=\frac{1}{2}(𝔼[W]+𝔼[W^{\prime }])=\frac{1}{2}(\tau (\theta +\theta )=\tau (\theta )$, $W^{*}$ is an unbiased estimator of $\tau (\theta )$.

Now, we consider the variance of $W^{*}$. $$\begin{array}{cc}\mathrm{Var}(W^{*})& =\frac{1}{4}\mathrm{Var}(W+W^{\prime })\\ & =\frac{1}{4}[\mathrm{Var}(W)+\mathrm{Var}(W^{\prime })+2\mathrm{Cov}(W,W^{\prime })]\\ & \le \frac{1}{4}\mathrm{Var}(W)+\frac{1}{4}\mathrm{Var}(W^{\prime })+\frac{1}{2}\sqrt{\mathrm{Var}(W)\mathrm{Var}(W^{\prime })}& (\text{covariance inequality})\\ & =\frac{1}{4}\mathrm{Var}(W)+\frac{1}{4}\mathrm{Var}(W)+\frac{1}{2}\sqrt{(\mathrm{Var}(W){)}^2}& (\mathrm{Var}(W)=\mathrm{Var}(W^{\prime })\text{ since }W\text{ and }{W}^{\prime }\text{ are both UMVUE})\\ & =\frac{1}{2}\mathrm{Var}(W)+\frac{1}{2}\mathrm{Var}(W)& (\mathrm{Var}(W)>0)\\ & =\mathrm{Var}(W).\end{array}$$ Thus, we now have either $\mathrm{Var}(W^{*})<\mathrm{Var}(W)$ or $\mathrm{Var}(W^{*})=\mathrm{Var}(W)$. If the former is true, then $W$ is Template:Colored em an UMVUE of $\tau (\theta )$ by definition, since we can find another unbiased estimator, namely $W^{*}$, with smaller variance than it. Hence, we must have the latter, i.e., $$\mathrm{Var}(W^{*})=\mathrm{Var}(W).$$ This implies when we apply the covariance inequality, the equality holds, i.e., $$\mathrm{Cov}(W,W^{\prime })=\sqrt{\mathrm{Var}(W)\mathrm{Var}(W^{\prime })}⟺\rho (W^{\prime },W)=1,$$ which means $W^{\prime }$ is increasing linearly with $W$, i.e., we can write $$W^{\prime }=aW+b$$ for some constants $a>0$ and $b$.

Now, we consider the covariance $\mathrm{Cov}(W,W^{\prime })$. $$\mathrm{Cov}(W,W^{\prime })\stackrel{\text{ above }}{=}\mathrm{Cov}(W,aW+b)\stackrel{\text{ properties }}{=}a\mathrm{Cov}(W,W)\stackrel{\text{ property }}{=}a\mathrm{Var}(W).$$ On the other hand, since the equality holds in the covariance inequality, and $\mathrm{Var}(W)=\mathrm{Var}(W^{\prime })$ (since they are both UMVUE), $$\mathrm{Cov}(W,W^{\prime })=\sqrt{\mathrm{Var}(W)\mathrm{Var}(W^{\prime })}=\sqrt{(\mathrm{Var}(W){)}^2}=\mathrm{Var}(W).$$ Thus, we have $a=1$.

It remains to show that $b=0$ to prove that $W=W^{\prime }$, and therefore conclude that $W$ is Template:Colored em.

From above, we currently have $W^{\prime }=W+b⟹𝔼[W^{\prime }]=𝔼[W]+b⟹\tau (\theta )=\tau (\theta )+b⟹b=0$, as desired.

$◻$

Cramer-Rao lower bound

Without using some results, it is quite difficult to determine the UMVUE, since there are many (perhaps even infinitely many) possible unbiased estimator, so it is quite hard to ensure that one particular unbiased estimator is relative more efficient than every other possible unbiased estimators.

Therefore, we will introduce some approaches that help us to find the UMVUE. For the first approach, we find a Template:Colored em on the variances of all possible unbiased estimators. After getting such lower bound, if we can find an unbiased estimator with variance to be exactly equal to the lower bound, then the lower bound is the minimum value of the variances, and hence such unbiased estimator is an UMVUE by definition. Template:Colored remark A common way to find such lower bound is to use the Template:Colored em (CRLB), and we get the CRLB through Template:Colored em. Before stating the inequality, let us define some related terms. Template:Colored definition Template:Colored remark For the regularity conditions which allow interchange of derivative and integral, they include

1. the partial derivatives involved should exist, i.e., the (natural log) of the functions involved is differentiable
2. the integrals involved should be differentiable
3. the support does not depend on the parameter(s) involved

We have some results that assist us to compute the Fisher information. Template:Colored proposition

Proof. $$\begin{array}{cc}{ℐ}_n(\theta )& =𝔼\left[{\left(\frac{\partial \ln ℒ(\mathit{\theta };𝐱)}{\partial \theta }\right)}^2\right]\\ & =\mathrm{Var}\left(\frac{\partial \ln ℒ(\mathit{\theta };𝐱)}{\partial \theta }\right)& \text{by above remark}\\ & =\mathrm{Var}\left(\frac{\partial }{\partial \theta }(\ln \prod _{i=1}^{n}f(X_i;\mathit{\theta }))\right)& (ℒ(\theta ;𝐱)=\prod _{i=1}^{n}f(x_i;\theta ))\\ & =\mathrm{Var}\left(\frac{\partial }{\partial \theta }(\sum _{i=1}^n\ln f(X_i;\mathit{\theta }))\right)\\ & =\mathrm{Var}(\sum _{i=1}^n\frac{\partial }{\partial \theta }\ln f(X_i;\mathit{\theta }))& \text{by linearity of differentiation}\\ & =\sum _{i=1}^n\mathrm{Var}(\frac{\partial }{\partial \theta }\ln f(X_i;\mathit{\theta }))& \text{by independence}\\ & =n\mathrm{Var}(\frac{\partial }{\partial \theta }\ln f(X_i;\mathit{\theta }))& \text{by identically distributed property}\\ & =n𝔼\left[{\left(\frac{\partial \ln f(X;\mathit{\theta })}{\partial \theta }\right)}^2\right]& \text{by above remark}\\ & =nℐ(\theta ).\end{array}$$

$◻$

Proof. $$\begin{array}{cc}𝔼\left[\frac{{\partial }^2\ln f(X;\mathit{\theta })}{\partial {\theta }^2}\right]& =𝔼\left[\frac{\partial }{\partial \theta }\left(\frac{\partial \ln f(X;\mathit{\theta })}{\partial \theta }\right)\right]\\ & =𝔼\left[\frac{\partial }{\partial \theta }(\frac{1}{f(X;\mathit{\theta })}\cdot \frac{\partial f(X;\mathit{\theta })}{\partial \theta })\right]\\ & =𝔼[\frac{1}{f(X;\mathit{\theta })}\cdot \frac{{\partial }^{2}f(X;\mathit{\theta })}{\partial {\theta }^2}-\frac{\partial f(X;\mathit{\theta })}{\partial \theta }\cdot \frac{1}{(f(X;\mathit{\theta }){)}^2}\cdot \frac{\partial f(X;\mathit{\theta })}{\partial \theta }]\\ & =𝔼[\frac{1}{f(X;\mathit{\theta })}\cdot \frac{{\partial }^{2}f(X;\mathit{\theta })}{\partial {\theta }^2}-{\left(\frac{\partial f(X;\mathit{\theta })}{\partial \theta }\right)}^2\cdot \frac{1}{(f(X;\mathit{\theta }){)}^2}]\\ & =𝔼[\frac{1}{f(X;\mathit{\theta })}\cdot \frac{{\partial }^{2}f(X;\mathit{\theta })}{\partial {\theta }^2}]-𝔼\left[{\left(\frac{\partial \ln f(X;\mathit{\theta })}{\partial \theta }\right)}^2\right]\\ & =𝔼[\frac{1}{f(X;\mathit{\theta })}\cdot \frac{{\partial }^{2}f(X;\mathit{\theta })}{\partial {\theta }^2}]-ℐ(\theta )\\ \end{array}$$ Now, it suffices to prove that $𝔼[\frac{1}{f(X;\mathit{\theta })}\cdot \frac{{\partial }^{2}f(X;\mathit{\theta })}{\partial {\theta }^2}]=0$, which is true since $$\begin{array}{cc}𝔼[\frac{1}{f(X;\mathit{\theta })}\cdot \frac{{\partial }^{2}f(X;\mathit{\theta })}{\partial {\theta }^2}]& =\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\frac{1}{f(x;\mathit{\theta })}\cdot \frac{{\partial }^{2}f(x;\mathit{\theta })}{\partial {\theta }^2}\cdot f(x;\mathit{\theta })dx\\ & =\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\frac{{\partial }^{2}f(x;\mathit{\theta })}{\partial {\theta }^2}dx\\ & =\frac{{\partial }^2}{\partial {\theta }^2}\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}f(x;\mathit{\theta })dx\\ & =\frac{{\partial }^2}{\partial {\theta }^2}(1)\\ & =0.\\ \end{array}$$

$◻$

Proof. Since $W$ is an unbiased estimator of $\tau (\theta )$, we have by definition $𝔼[W]=\tau (\theta )$. By definition of expectation, we have $𝔼[W]=\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\cdots \underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}wℒ(\theta ;𝐱)d{x}_n\cdots d{x}_1$ where $ℒ(\theta ;𝐱)$ is the likelihood function. Thus, $$\begin{array}{cccc}& & \underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\cdots \underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}wℒ(\theta ;𝐱)d{x}_n\cdots d{x}_1& =\tau (\theta )\\ & \Rightarrow & \frac{\partial }{\partial \theta }\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\cdots \underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}wℒ(\theta ;𝐱)d{x}_n\cdots d{x}_1& =\frac{\partial }{\partial \theta }\tau (\theta )\\ & \Rightarrow & \underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\cdots \underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\frac{\partial }{\partial \theta }(wℒ(\theta ;𝐱))d{x}_n\cdots d{x}_1& ={\tau }^{\prime }(\theta )\\ & \Rightarrow & \underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\cdots \underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}w\frac{\partial }{\partial \theta }(ℒ(\theta ;𝐱))\cdot \frac{1}{ℒ(\theta ;𝐱)}\cdot ℒ(\theta ;𝐱)d{x}_n\cdots d{x}_1& ={\tau }^{\prime }(\theta )\\ & \Rightarrow & \underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\cdots \underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}w\frac{\partial \ln ℒ(\theta ;𝐱)}{\partial \theta }ℒ(\theta ;𝐱)d{x}_n\cdots d{x}_1& ={\tau }^{\prime }(\theta )\\ & \Rightarrow & 𝔼[W\cdot \frac{\partial \ln ℒ(\theta ;𝐱)}{\partial \theta }]& ={\tau }^{\prime }(\theta )\\ & \Rightarrow & 𝔼[WS(\theta ;𝐗)]& ={\tau }^{\prime }(\theta )& (S(\theta ;𝐗)=\frac{\partial \ln ℒ(\theta ;𝐱)}{\partial \theta })\\ & \Rightarrow & 𝔼[WS(\theta ;𝐗)]-𝔼[W]\underset{=0}{\underbrace{𝔼[S(\theta ;𝐗)]}}& ={\tau }^{\prime }(\theta )& (𝔼[S(\theta ;𝐗)]=0\text{ by remark about Fisher information})\\ & \Rightarrow & \mathrm{Cov}(W,S(\theta ;𝐗))& ={\tau }^{\prime }(\theta )\\ \end{array}$$ Consider the covariance inequality: $(\mathrm{Cov}(X,Y){)}^2\le \mathrm{Var}(X)\mathrm{Var}(Y)$. We have $$(\mathrm{Cov}(W,S(\theta ;𝐗)){)}^2\le \mathrm{Var}(W)\mathrm{Var}(S(\theta ;𝐗))⟹({\tau }^{\prime }(\theta ){)}^2\le \mathrm{Var}(W)\mathrm{Var}(S(\theta ;𝐗))⟹\mathrm{Var}(W)\ge \frac{({\tau }^{\prime }(\theta ){)}^2}{\mathrm{Var}(S(\theta ;𝐗))}=\frac{({\tau }^{\prime }(\theta ){)}^2}{{ℐ}_n(\theta )}.$$ (${ℐ}_n(\theta )=\mathrm{Var}(S(\theta ;𝐗))$ by remark about Fisher information)

$◻$

Template:Colored remark Template:Colored example Sometimes, we cannot use the CRLB method for finding UMVUE, because

• the regularity conditions may not be satisfied, and thus we cannot use the Cramer-Rao inequality, and
• the variance of the unbiased estimator may not be equal to the CRLB, but we cannot conclude that it is not an UMVUE, because it may be the case that the CRLB is not attainable at all, and the smallest variance among all unbiased estimators is actually the variance of that estimator, which is larger than the CRLB.

We will illustrate some examples for these two cases in the following. Template:Colored example Template:Colored example Template:Colored remark Since the CRLB is sometimes attainable and sometimes not, it is natural to question that Template:Colored em can the CRLB be attained. In other words, we would like to know the Template:Colored em for the CRLB, which are stated in the following corollary. Template:Colored corollary

Proof. Considering the proof for Cramer-Rao inequality, we have $$\mathrm{Var}(W)=\frac{({\tau }^{\prime }(\theta ){)}^2}{{ℐ}_n(\theta )}⟺(\mathrm{Cov}(W,S(\theta ;𝐗)){)}^2=\mathrm{Var}(W)\mathrm{Var}(S(\theta ;𝐗))$$ We can write $\mathrm{Cov}(W,S(\theta ;𝐗))$ as $\mathrm{Cov}(W-\underset{\text{constant}}{\underbrace{\tau (\theta )}},S(\theta ;𝐗))$ (by result about covariance). Also, $\mathrm{Var}(W)=\mathrm{Var}(W-\underset{\text{constant}}{\underbrace{\tau (\theta )}})$ (by result about variance). Thus, we have $$\begin{array}{cccc}& & (\mathrm{Cov}(W-\tau (\theta ),S(\theta ;𝐗)){)}^2& =\mathrm{Var}(W-\tau (\theta ))\mathrm{Var}(S(\theta ;𝐗))\\ & \iff & \frac{(\mathrm{Cov}(W-\tau (\theta ),S(\theta ;𝐗)){)}^2}{\mathrm{Var}(W-\tau (\theta ))\mathrm{Var}(S(\theta ;𝐗))}& =1\\ & \iff & \frac{(\mathrm{Cov}(S(\theta ;𝐗),W-\tau (\theta )){)}^2}{\mathrm{Var}(W-\tau (\theta ))\mathrm{Var}(S(\theta ;𝐗))}& =1\\ & \iff & (\rho (S(\theta ;𝐗),W-\tau (\theta )){)}^2& =1\\ & \iff & \rho (S(\theta ;𝐗),W-\tau (\theta ))& =\pm 1\end{array}$$ where $\rho (\cdot ,\cdot )$ is the correlation coefficient between two random variables. This means $S(\theta ;𝐗)$ increases or decreases linearly with $W-\tau (\theta )$, i.e., $$S(\theta ;𝐗)=k(W-\tau (\theta ))+c$$ for some constants $c,k$. Now, it suffices to show that the constant $c$ is actually zero.

We know that $𝔼[W]=\tau (\theta )$ (since $W$ is an unbiased estimator of $\tau (\theta )$), and $𝔼[S(\theta ;𝐗)]=0$ (from remark about Fisher information). Thus, applying expectations on both side gives $$𝔼[S(\theta ;𝐗)]=k𝔼[W-\tau (\theta )]+c⟺𝔼[S(\theta ;𝐗)]=k(\underset{=0}{\underbrace{𝔼[W]-\tau (\theta )}})+c⟺0=0+c⟺c=0.$$ Then, the result follows.

$◻$

Template:Colored remark Template:Colored example Template:Colored remark Template:Colored example Template:Colored remark We have discussed MLE previously, and MLE is actually a "best choice" asymptotically (i.e., as the sample size $n\to \mathrm{\infty }$) according to the following theorem. Template:Colored theorem

Proof. Template:Colored em: we consider the Taylor series of order 2 for $\frac{d}{d\theta }\ln ℒ(\theta )$, and we will get $$\frac{d}{d\theta }\ln ℒ(\hat{\theta })=\frac{d}{d\theta }\ln ℒ(\theta )+(\hat{\theta }-\theta )\frac{d^2}{d{\theta }^2}\ln ℒ(\theta )+\frac{1}{2}(\hat{\theta }-\theta {)}^2\frac{d^3}{d{\theta }^3}\ln ℒ(\theta ){|}_{\theta ={\theta }^{*}}$$ where ${\theta }^{*}$ is between $\theta$ and $\hat{\theta }$. Since $\hat{\theta }$ is the MLE of $\theta$, from the derivative test, we know that $\frac{d}{d\theta }\ln ℒ(\hat{\theta })=0$ (we apply regularity condition to ensure the existence of this derivative). Hence, we have $$\begin{array}{cccc}& & \frac{d}{d\theta }\ln ℒ(\theta )+(\hat{\theta }-\theta )\frac{d^2}{d{\theta }^2}\ln ℒ(\theta )+\frac{1}{2}(\hat{\theta }-\theta {)}^2\frac{d^3}{d{\theta }^3}\ln ℒ(\theta ){|}_{\theta ={\theta }^{*}}& =0\\ & \Rightarrow & -\sqrt{n}(\hat{\theta }-\theta )\frac{d^2}{d{\theta }^2}\ln ℒ(\theta )-\frac{\sqrt{n}}{2}(\hat{\theta }-\theta {)}^2\frac{d^3}{d{\theta }^3}\ln ℒ(\theta ){|}_{\theta ={\theta }^{*}}=\sqrt{n}\frac{d}{d\theta }\ln ℒ(\theta )\\ & \Rightarrow & \sqrt{n}(\hat{\theta }-\theta )=\frac{\frac{d}{d\theta }\ln ℒ(\theta )/\sqrt{n}}{-n^{-1}\frac{d^2}{d{\theta }^2}\ln ℒ(\theta )-(2n{)}^{-1}(\hat{\theta }-\theta )\frac{d^3}{d{\theta }^3}\ln ℒ(\theta ){|}_{\theta ={\theta }^{*}}}.\end{array}$$ Since $$\mathrm{Var}\left(\sum _{i=1}^n\frac{\partial \ln f(X_i;\theta )}{\partial \theta }\right)=\sum _{i=1}^n\mathrm{Var}\left(\frac{\partial \ln f(X_i;\theta )}{\partial \theta }\right)=\sum _{i=1}^n𝔼\left[{\left(\frac{\partial \ln f(X_i;\theta )}{\partial \theta }\right)}^2\right]=nℐ(\theta )(1),$$ by central limit theorem, $$\frac{\frac{d}{d\theta }\ln ℒ(\theta )}{\sqrt{n}}=\frac{1}{\sqrt{n}}\sum _{i=1}^n\frac{\partial \ln f(X_i;\theta )}{\partial \theta }\stackrel{d}{\to }𝒩(0,(1/n)nI(\theta ))\equiv 𝒩(0,ℐ(\theta )).$$ Furthermore, we apply the weak law of large number to show that $$-n^{-1}\frac{d^2}{d{\theta }^2}\ln ℒ(\theta )=-\frac{1}{n}\sum _{i=1}^n\frac{{\partial }^2\ln f(X_i;\theta )}{\partial {\theta }^2}\stackrel{p}{\to }-𝔼\left[\frac{{\partial }^2\ln f(X_i;\theta )}{\partial {\theta }^2}\right]=ℐ(\theta )(2).$$ It can be shown in a quite complicated way (and using regularity conditions) that $$-(2n{)}^{-1}(\hat{\theta }-\theta )\frac{d^3}{d{\theta }^3}\ln ℒ(\theta ){|}_{\theta ={\theta }^{*}}\stackrel{p}{\to }0.(3).$$ Considering $(2)$ and $(3)$, using property of convergence in probability, we have $$-n^{-1}\frac{d^2}{d{\theta }^2}\ln ℒ(\theta )-(2n{)}^{-1}(\hat{\theta }-\theta )\frac{d^3}{d{\theta }^3}\ln ℒ(\theta ){|}_{\theta ={\theta }^{*}}\stackrel{p}{\to }ℐ(\theta )+0=ℐ(\theta )(4).$$ Considering $(1)$ and $(4)$, and using Slutsky's theorem, we have $$\sqrt{n}(\hat{\theta }-\theta )=\frac{\frac{d}{d\theta }\ln ℒ(\theta )/\sqrt{n}}{-n^{-1}\frac{d^2}{d{\theta }^2}\ln ℒ(\theta )-(2n{)}^{-1}(\hat{\theta }-\theta )\frac{d^3}{d{\theta }^3}\ln ℒ(\theta ){|}_{\theta ={\theta }^{*}}}\stackrel{d}{\to }\frac{Y}{ℐ(\theta )}$$ where $Y\sim 𝒩(0,ℐ(\theta ))$, and hence $\frac{Y}{ℐ(\theta )}\sim 𝒩(0,\frac{ℐ(\theta )}{[ℐ(\theta ){]}^2})\equiv 𝒩(0,1/ℐ(\theta ))$. It follows that $$\sqrt{n}(\hat{\theta }-\theta )\stackrel{d}{\to }𝒩(0,1/ℐ(\theta )).$$ This means $$\hat{\theta }-\theta \stackrel{d}{\to }𝒩(0,1/(nℐ(\theta )))\equiv 𝒩(0,1/{ℐ}_n(\theta )),$$ and thus $$\frac{\hat{\theta }-\theta }{\sqrt{1/{ℐ}_n(\theta )}}\stackrel{d}{\to }𝒩(0,\frac{1/(nℐ(\theta ))}{1/\underset{=nℐ(\theta )}{\underbrace{{ℐ}_n(\theta )}}})\equiv 𝒩(0,1)$$ as desired.

$◻$

Template:Colored remark Since we are not able to use the CRLB to find UMVUE in some situations, we will introduce another method to find UMVUE in the following, which uses the concepts of Template:Colored em and Template:Colored em.

Sufficiency

Intuitively, a Template:Colored em $T(X_1,\dots ,X_n)$, which is a function of a given random sample $X_1,\dots ,X_n$, contains all information needed for estimating the unknown parameter (vector) $\theta$. Thus, the statistic $T(X_1,\dots ,X_n)$ itself is "sufficient" for estimating the unknown parameter (vector) $\theta$.

Formally, we can define and describe Template:Colored em as follows: Template:Colored definition Template:Colored remark Template:Colored example Template:Colored remark Let us state the above remark about transformation of sufficient statistic formally below. Template:Colored proposition Now, we discuss a theorem that helps us to check the sufficiency of a statistic, namely (Fisher-Neyman) Template:Colored em. Template:Colored theorem

Proof. Since the proof for continuous case is quite complicated, we will only give a proof for the discrete case. For simplicity of presentation, let $𝐗=(X_1,\dots ,X_n)$, $T=T(X_1,\dots ,X_n)$, $𝐱=(x_1,\dots ,x_n)$, and $t=T(x_1,\dots ,x_n)$, and hence there are notations for different types of pmfs from these. By definition, $f_{𝐗|T}(𝐱|t;\theta )=f_{𝐗|T}(𝐱,t)$. Also, we have $𝐗=𝐱⟺𝐗=𝐱\cap T(𝐗)=T(𝐱)⟺𝐗=𝐱\cap T=t$. Thus, we can write $$f_{𝐗,T}(𝐱,t;\theta )=f_{𝐗}(𝐱;\theta )(*)$$.

"only if" ($\Rightarrow$) direction: Assume $T$ is a sufficient statistic. Then, we choose $g(t;\theta )=f_T(t;\theta )$ and $h(𝐱)=f_{𝐗|T}(𝐱|t)$, which does not depend on $\theta$ by the definition of sufficient statistic. It remains to verify that the equation actually holds for this choice.

Hence, $$f_{𝐗}(𝐱;\theta )=f_{𝐗,T}(𝐱,t;\theta )\stackrel{\text{ def }}{=}{f}_{𝐗|T}(𝐱|t;\theta )f_T(t;\theta )\stackrel{\text{ sufficiency }}{=}{f}_{𝐗|T}(𝐱|t)f_T(t;\theta )=h(𝐱)g(t;\theta ).$$

"if" ($\Leftarrow$) direction: Assume we can write $f_{𝐗}(𝐱;\theta )=g(t;\theta )h(𝐱)$. Then, $$f_T(t;\theta )\stackrel{\text{ marginal pmf }}{=}\sum _{𝐱}^{}{f}_{𝐗,T}(𝐱,t;\theta )\stackrel{\text{ (*) }}{=}\sum _{𝐱}^{}{f}_{𝐗}(𝐱;\theta )\stackrel{\text{ assumption }}{=}\sum _{𝐱}^{}g(t;\theta )h(𝐱)=\underset{\text{independent from }𝐱}{\underbrace{g(t;\theta )}}\sum _{𝐱}^{}h(𝐱).$$ Now, we aim to show that $f_{𝐗|T}(𝐱|t)$ does not depend on $\theta$, which means $T$ is a sufficient statistic for $\theta$. We have $$f_{𝐗|T}(𝐱|t)\stackrel{\text{ def }}{=}\frac{f_{𝐗,T}(𝐱,t;\theta )}{f_T(t;\theta )}\stackrel{\text{ (*) }}{=}\frac{f_{𝐗}(𝐱;\theta )}{f_T(t;\theta )}=\frac{\stackrel{\text{assumption}}{\overbrace{g(t;\theta )h(𝐱)}}}{\underset{\text{above}}{\underbrace{g(t;\theta )\sum _{𝐱}^{}h(𝐱)}}}=\frac{h(𝐱)}{\sum _{𝐱}^{}h(𝐱)},$$ which does not depend on $\theta$, as desired.

$◻$

Template:Colored remark Template:Colored example For some "nice" distributions, which belong to Template:Colored em, sufficient statistics can be found using another alternative method easily and more conveniently. This method works because of the "nice" form of the pdf or pmf of those distributions, which can be characterized as follows: Template:Colored definition Template:Colored remark Template:Colored example Template:Colored theorem

Proof. Since the distribution belongs to the exponential family, the joint pdf or pmf of $X_1,\dots ,X_n$ can be expressed as $$\begin{array}{cc}f(x_1,\dots ,x_n;\theta )& =\prod _{j=1}^n[h(x_j)g(\theta )\exp (\sum _{i=1}^s{\eta }_i(\theta )T_i(x_j))]\\ & =[\prod _{j=1}^{n}h(x_j)](g(\theta ){)}^n\exp (\sum _{j=1}^n\sum _{i=1}^s{\eta }_i(\theta )T_i(x_j))\\ & =[\prod _{j=1}^{n}h(x_j)](g(\theta ){)}^n\exp (\sum _{i=1}^s\sum _{j=1}^n{\eta }_i(\theta )T_i(x_j))& (\text{changing summation order, where the upper bounds are constants})\\ & =[\prod _{j=1}^{n}h(x_j)](g(\theta ){)}^n\exp (\sum _{i=1}^s\underset{\text{independent from }j}{\underbrace{{\eta }_i(\theta )}}\sum _{j=1}^n{T}_i(x_j))\\ & =[\prod _{j=1}^{n}h(x_j)](g(\theta ){)}^n\exp ({\eta }_1(\theta )\sum _{j=1}^n{T}_1(x_j)+\cdots +{\eta }_s(\theta )\sum _{j=1}^n{T}_s(x_j)).\\ \end{array}$$ From here, for applying the factorization theorem, we can identify the Template:Color part of the function as "$h(x_1,\dots ,x_n)$", and the Template:Color part of the function as "$g(T(x_1,\dots ,x_n);\theta )$". We can notice that the Template:Color part of the function depends on $x_1,\dots ,x_n$ only through $(\sum _{j=1}^n{T}_1(x_j),\dots ,\sum _{j=1}^n{T}_s(x_j))$. The result follows.

$◻$

Template:Colored example Now, we will start discussing how is sufficient statistic related to UMVUE. We begin our discussion by Template:Colored em. Template:Colored theorem

Proof. Assume $W$ is an arbitrary unbiased estimator of $\tau (\theta )$, and $T$ is a sufficient statistic of $\theta$.

First, we prove that $\phi (T)$ is an unbiased estimator of $\tau (\theta )$. Before proving the unbiasedness, we should ensure that $\phi (T)$ is actually an estimator, i.e., it is a statistic, which is a function of random sample, and needs to be independent from $\theta$ (so that it is calculable): since $W$ is a function of random sample, and $T$ is a sufficient statistic, which make the conditional distribution of $W$, given $T$, Template:Colored em of $\theta$. Also, $\phi (T)=𝔼[W|T]$ is a function of $W$, and thus is also a function of random sample.

Now, we prove that $\phi (T)$ is an unbiased estimator of $\tau (\theta )$: since $𝔼[\phi (T)]=𝔼[𝔼[W|T]]\stackrel{\text{ law of total expectation }}{=}𝔼[W]\stackrel{\text{ unbiasedness }}{=}\tau (\theta )$, $\phi (T)$ is an unbiased estimator of $\tau (\theta )$.

Next, we prove that $\mathrm{Var}(\phi (T))\le \mathrm{Var}(W)$: by law of total variance, we have $$\mathrm{Var}(W)=\mathrm{Var}(𝔼[W|T])+𝔼[\mathrm{Var}(W|T)]\stackrel{\text{ def }}{=}\mathrm{Var}(\phi (T))+\stackrel{\ge 0}{\overbrace{𝔼[\underset{\ge 0}{\underbrace{\mathrm{Var}(W|T)}}]}}\ge \mathrm{Var}(\phi (T)),$$ as desired.

$◻$

Template:Colored remark To actually determine the UMVUE, we need another theorem, called Template:Colored em, which is based on Rao-Blackwell theorem, and requires the concept of Template:Colored em.

Completeness

Template:Colored definition When a random sample $X_1,\dots ,X_n$ is from a distribution in exponential family, then a complete statistic can also be founded easily, similar to the case for sufficient statistic. Template:Colored theorem

Proof. Omitted.

$◻$

Proof. Assume $T$ is a Template:Colored em for $\theta$ and $𝔼[\phi (T)]=\tau (\theta )$.

Since $T$ is a sufficient statistic for $\theta$, we can apply the Rao-Blackwell theorem. From Rao-Blackwell theorem, if $W$ is an arbitrary unbiased estimator of $\tau (\theta )$, then $\phi (T)$ is another unbiased estimator where $\mathrm{Var}(\phi (T))\le \mathrm{Var}(W)$.

To prove that $\phi (T)$ is the unique UMVUE of $\tau (\theta )$, we proceed to show that Template:Colored em of the choice of the unbiased estimator $W$ of $\tau (\theta )$, we get the Template:Colored em $\phi (T)$ from the Rao-Blackwell theorem (with probability 1). Then, we will have Template:Colored em possible unbiased estimator $W$ of $\tau (\theta )$, $\mathrm{Var}(\phi (T))\le \mathrm{Var}(W)$ (with probability 1) , which means $\phi (T)$ is the UMVUE, and is also the Template:Colored em UMVUE since we always get the same $\phi (T)$ .

Assume that $W^{\prime }$ is Template:Colored em unbiased estimator of $\tau (\theta )$ ($W^{\prime }\ne W$). By Rao-Blackwell theorem again, there is an unbiased estimator $\psi (T)=𝔼[W^{\prime }|T]$ ($\psi (T)\ne \phi (T)$) where $\mathrm{Var}(\psi (T))\le \mathrm{Var}(W^{\prime })$. Since both $\phi (T)$ and $\psi (T)$ are unbiased estimators of $\tau (\theta )$, we have for each $\theta \in \mathrm{\Theta }$, $$𝔼[\phi (T)]=𝔼[\psi (T)]⟹𝔼[\phi (T)-\psi (T)]=0.$$ Since $T$ is a complete statistic, we have $$\mathbb{P}(\phi (T)-\psi (T)=0)=1⟹\mathbb{P}(\phi (T)=\psi (T))=1,$$ which means $\phi (T)=\psi (T)$ (with probability 1), i.e., we get the same $\phi (T)$ from the Rao-Blackwell theorem in this case (with probability 1).

$◻$

Consistency

In the previous sections, we have discussed Template:Colored em and Template:Colored em. In this section, we will discuss another property called Template:Colored em. Template:Colored definition Template:Colored remark Template:Colored proposition

Proof. Assume $\hat{\theta }$ is an (asymptotically) unbiased estimator of an unknown parameter $\theta$ and $\mathrm{Var}(\hat{\theta })\to 0$ as $n\to \mathrm{\infty }$. Since $\hat{\theta }$ is an (asymptotically) unbiased estimator of $\theta$, we have $\underset{n\to \mathrm{\infty }}{\lim }\mathrm{Bias}(\hat{\theta })=0$ (this is true for both asymptotically unbiased estimator and unbiased estimator of $\theta$). In addition to this, we have by assumption that $\underset{n\to \mathrm{\infty }}{\lim }\mathrm{Var}(\hat{\theta })=0$. By definition of mean squared error, these imply that $\underset{n\to \mathrm{\infty }}{\lim }\mathrm{MSE}(\hat{\theta })=0\Rightarrow \underset{n\to \mathrm{\infty }}{\lim }𝔼[(\hat{\theta }-\theta {)}^2]=0$. Thus, as $n\to \mathrm{\infty }$, we have by Chebyshov's inequality (notice that $\mathrm{MSE}(\hat{\theta })=𝔼[(\hat{\theta }-\theta {)}^2]$ exist from above), for each $\epsilon >0$, $$\mathbb{P}(|\hat{\theta }-\theta |>\epsilon )\le \frac{𝔼[(\hat{\theta }-\theta {)}^2]}{{\epsilon }^2}\to \frac{0}{{\epsilon }^2}=0.$$ Since probability is nonnegative ($\ge 0$), and this probability is less than or equal to an expression that tends to be 0 as $n\to \mathrm{\infty }$, we conclude that this probability tends to be zero as $n\to \mathrm{\infty }$. That is, $\hat{\theta }$ is a Template:Colored em of $\theta$.

$◻$