Statistics/Interval Estimation

Introduction

Previously, we have discussed Template:Colored em, which gives us an estimator $\hat{\theta }$ for the value of an unknown parameter $\theta$. Now, suppose we want to know the size of Template:Colored em of the point estimator $\hat{\theta }$, i.e. the difference between $\hat{\theta }$ and the unknown parameter $\theta$. Of course, we can make use of the value of the Template:Colored em of $\hat{\theta }$, $𝔼[(\hat{\theta }-\theta {)}^2]$, or other things.

However, what if we only know about one specific Template:Colored em? We cannot calculate the mean squared error of its corresponding point estimator with Template:Colored em this point estimates, right? So, how do we know the possible size of error of this Template:Colored em? Indeed, it is impossible to tell, since we are only given a particular estimated value of parameter $\theta$, but of course we do know the value of the unknown parameter $\theta$, thus the difference between this point estimate and $\theta$ is also unknown.

To illustrate this, consider the following example: suppose we take a random sample of 10 students from one particular course in university to estimate the mean score of the students in the final exam in that course, denoted by $\mu$, (assume the score is normally distributed), and the observed value of the sample mean is $\bar{x}=60$. Then, what is the difference between this point estimate and the true unknown parameter $\mu$? Can we be "confident" that this sample mean is close to $\mu$, say $\mu \in [\bar{x}-5,\bar{x}+5]=[55,65]$?

It is possible that $\mu$ is, say 90, and somehow the students in the sample are the one with very poor performance. On the other hand, it is also possible that $\mu$ is, say 30, and somehow the students in the sample are the one who perform well (relatively). Of course, it is also possible that the $\mu$ is quite close to 60, say 59. From this example, we can see that a particular value $\bar{x}=60$ does not tell us the possible size of error: the error can be very large, and also can be very small.

In this chapter, we will introduce Template:Colored em where we use Template:Colored em that can describe the size of error through providing the probability for the random interval (i.e. interval with at least one of its bounds to be a random variable) given by the interval estimator to contain the unknown parameter $\theta$, which measures the "accuracy" of the interval estimator of $\theta$, and hence the size of error.

As suggested by the name Template:Colored em, the estimator involves some sort of Template:Colored em. Also, as one may expect, Template:Colored em is also based on Template:Colored em: Template:Colored definition Of course, we would like the probability for the unknown parameter $\theta$ to lie in the interval to be close to 1, so that the interval estimator is very accurate. However, a very accurate interval estimator may have a very bad "precision", i.e. the interval covers "too many" plausible values of an unknown parameter, and therefore even if we know that $\theta$ is very likely to be one of such values, there are too many different possibilities. Hence, such interval estimator is not very "useful". To illustrate this, suppose the interval concerned is $ℝ$, which is the parameter space of $\theta$. Then, of course $\mathbb{P}(\theta \in ℝ)=1$ (so the "confidence" is high) since $\theta$ must lie in its parameter space. However, such interval has basically "zero precision", and is quite "useless", since the "plausible values" of $\theta$ in the intervals are essentially all possible values of $\theta$.

From this, we can observe the need of the "precision" of the interval, that is, we also want the Template:Colored em of the interval to be small, so that we can have some ideas about the "location" of $\theta$. However, as the interval becomes smaller, it is more likely that such interval misses $\theta$, i.e. does not cover the actual value of $\theta$, and therefore the probability for $\theta$ to lie in that interval becomes smaller, i.e. the interval becomes less "accurate". To illustrate this, let us consider the extreme case: the interval is so small that it becomes an interval containing a single point (the two end-points of the interval coincide). Then, the "interval estimator" basically becomes a "point estimator" in some sense, and we know that it is very unlikely that the true value of $\theta$ equals the value of the point estimator $\hat{\theta }$ ($\theta$ lies in that "interval" is equivalent to $\theta =\hat{\theta }$ in this case). Indeed, if the distribution of $\hat{\theta }$ is Template:Colored em, then $\mathbb{P}(\hat{\theta }=\theta )=0$.

As we can see from above, although we want the interval to have a very high "confidence" and also "very precise" (i.e. the interval is very narrow), we cannot have both of them, since an increase in confidence causes a decrease in "precision", and an increase in "precision" causes a decrease in confidence. Therefore, we need to make some compromises between them, and pick an interval that gives a sufficiently high confidence, and also is quite precise. In other words, we would like to have a Template:Colored em interval that will cover $\theta$ with a Template:Colored em.

Terminologies

Now, let us formally define some terminologies related to Template:Colored em. Template:Colored definition Template:Colored remark Template:Colored definition Template:Colored example Template:Colored definition Template:Colored remark Template:Colored example Template:Colored remark Template:Colored example Template:Colored exercise

Construction of confidence intervals

After understanding what confidence interval is, we would like to know how to construct one naturally. A main way for such construction is using the Template:Colored em, which is defined below. Template:Colored definition Template:Colored remark After having such pivotal quantity $Q(𝐗,\theta )$, we can construct a $1-\alpha$ confidence interval for $\theta$ by the following steps:

1. For that value of $\alpha$, find $a,b$ such that $\mathbb{P}(a\le Q(𝐗,\theta )\le b)=1-\alpha$ ($a,b$ does not involve $\theta$ since $Q(𝐗,\theta )$ is a pivotal quantity).
2. After that, we can transform $a\le Q(𝐗,\theta )\le b$ to $L(𝐗)\le \theta \le U(𝐗)$ since the expression of $Q(𝐗,\theta )$ involves $\theta$, as we have assumed (the resulting inequalities should be Template:Colored em to the original inequalities, that is, $a\le Q(𝐗,\theta )\le b⟺L(𝐗)\le \theta \le U(𝐗)$, so that $\mathbb{P}(L(𝐗)\le \theta \le U(𝐗))=\mathbb{P}(a\le Q(𝐗,\theta )\le b)$).

Confidence intervals for means of normal distributions

In the following, we will use the concept of pivotal quantity to construct confidence intervals for means and variances of Template:Colored em distributions. After that, because of the central limit theorem, we can construct Template:Colored em confidence intervals for means and variances of other types of distributions that are not normal.

Mean of a normal distribution

Before discussing this confidence interval, let us first introduce a notation:

• $z_{\alpha }$ is the upper percentile of $𝒩(0,1)$ at level $\alpha$, i.e. it satisfies $\mathbb{P}(Z\ge z_{\alpha })=\alpha$ where $Z\sim 𝒩(0,1)$.

We can find (or calculate) the values of $z_{\alpha }$ for different $\alpha$ from Template:Colored em. Template:Colored theorem Template:Colored remark

Proof. Let $Z=\frac{\bar{X}-\mu }{\sigma /\sqrt{n}}\sim 𝒩(0,1)$. Since $Z$ is a pivotal quantity (its distribution is independent from $\mu$), we set $$1-\alpha =1-\mathbb{P}(Z\ge z_{\alpha /2})-\mathbb{P}(Z\le -z_{\alpha /2})=\mathbb{P}(-z_{\alpha /2}<Z<z_{\alpha /2})=\mathbb{P}(-z_{\alpha /2}\le Z\le z_{\alpha /2}),$$ where $z_{\alpha /2}$ is a constant (and does not involve $\mu$). Then, we have $$\begin{array}{cc}1-\alpha & =\mathbb{P}(-z_{\alpha /2}\le Z\le z_{\alpha /2})\\ & =\mathbb{P}(-z_{\alpha /2}\le \frac{\bar{X}-\mu }{\sigma /\sqrt{n}}\le z_{\alpha /2})\\ & =\mathbb{P}(-z_{\alpha /2}\frac{\sigma }{\sqrt{n}}\le \bar{X}-\mu \le z_{\alpha /2}\frac{\sigma }{\sqrt{n}})\\ & =\mathbb{P}(z_{\alpha /2}\frac{\sigma }{\sqrt{n}}\ge \mu -\bar{X}\ge -z_{\alpha /2}\frac{\sigma }{\sqrt{n}})\\ & =\mathbb{P}(-z_{\alpha /2}\frac{\sigma }{\sqrt{n}}\le \mu -\bar{X}\le z_{\alpha /2}\frac{\sigma }{\sqrt{n}})& (\text{rewrite})\\ & =\mathbb{P}(\bar{X}-z_{\alpha /2}\frac{\sigma }{\sqrt{n}}\le \mu \le \bar{X}+z_{\alpha /2}\frac{\sigma }{\sqrt{n}}).\\ \end{array}$$ The result follows.

$◻$

The following graph illustrates $\mathbb{P}(-z_{\alpha /2}\le Z\le z_{\alpha /2})=1-\alpha$:

                    |
                  *-|-*
                 /##|##\   
                /###|###\  <----- area 1-a
               /####|####\
              /#####|#####\
             /######|######\
            /|######|######|\
 area    --*.|######|######|.*-- 
 a/2 --> ....|######|######|....  <---  area a/2
        ------------*---------------
           -z_{a/2}       z_{a/2}

Template:Colored example Template:Colored example We have previously discussed a way to construct confidence interval for the mean when the variance is Template:Colored em. However, this is not always the case in practice. We may not know the variance, right? Then, we cannot use the $\sigma$ in the confidence interval from the previous theorem.

Intuitively, one may think that we can use the Template:Colored em $S^2$ to "replace" the ${\sigma }^2$, according to the weak law of large number. Then, we can simply replace the unknown $\sigma$ in the confidence interval by the known $S$ (or its realization $s$ for interval estimate). However, the flaw in this argument is that the sample size may not be large enough to apply the weak law of large number for approximation. Template:Colored remark So, you may now ask that when the sample size is large enough, can we do such "replacement" for approximation. The answer is Template:Colored em, and we will discuss in the last section about approximated confidence intervals.

Before that section, the confidence intervals discussed is Template:Colored em in the sense that no approximation is used to construct them. Therefore, the confidence intervals constructed "work" for Template:Colored em sample size, no matter how large or how small it is (it works even if the sample size is 1, although such confidence interval constructed may not be very "nice", in the sense that the width of the interval may be quite large).

Before discussing how to construct an confidence interval for the mean when the variance is unknown, we first give some results that are useful for deriving such confidence interval.

Proof.

(i) One may use Basu's theorem to prove this, but the details about Basu's theorem and the proof are omitted here, since they are a bit complicated.

(ii) We will use the following definition of chi-squared distribution ${\chi }_k^2$ : $\sum _{i=1}^k{Z}_i^2\sim {\chi }_k^2$ where $Z_1,Z_2,\dots ,Z_k\sim 𝒩(0,1)$ are independent. Also, we will use the fact that the mgf of ${\chi }_k^2$ is $M(t)=(1-2t{)}^{-k/2},t<\frac{1}{2}$.

Now, first let $W=\sum _{i=1}^n{\left(\frac{X_i-\mu }{\sigma }\right)}^2$ which follows ${\chi }_n^2$ since $\frac{X_1-\mu }{\sigma },\dots ,\frac{X_n-\mu }{\sigma }\sim 𝒩(0,1)$ are independent. Then, we write $W$ as $$\begin{array}{cc}W& =\sum _{i=1}^n{\left(\frac{X_i-\mu }{\sigma }\right)}^2\\ & =\sum _{i=1}^n{(\frac{X_i-\bar{X}}{\sigma }+\frac{\bar{X}-\mu }{\sigma })}^2\\ & =\sum _{i=1}^n{\left(\frac{X_i-\bar{X}}{\sigma }\right)}^2+\sum _{i=1}^n{\left(\frac{\bar{X}-\mu }{\sigma }\right)}^2+0& (2\sum _{i=1}^n\frac{(\bar{X}-\mu )(X_i-\bar{X})}{{\sigma }^2}=\frac{2(\bar{X}-\mu )}{{\sigma }^2}(\underset{=n\bar{X}}{\underbrace{\sum _{i=1}^n{X}_i}}-\underset{=n\bar{X}}{\underbrace{\sum _{i=1}^n\stackrel{\text{constant wrt }i}{\overbrace{\bar{X}}}}})=0)\\ & =\frac{1}{{\sigma }^2}\underset{=n{S}^2}{\underbrace{\sum _{i=1}^n(X_i-\bar{X}{)}^2}}+(\underset{\sim 𝒩(0,1)\text{ by property}}{\underbrace{\frac{\sqrt{n}(\bar{X}-\mu )}{\sigma }}}{)}^2\\ & =\frac{n{S}^2}{{\sigma }^2}+Z^2& (Z=\frac{\sqrt{n}(\bar{X}-\mu )}{\sigma }\sim 𝒩(0,1))\end{array}$$ Applying the definition of chi-squared distribution, we have $Z^2\sim {\chi }_1^2$.

By (i), $\bar{X}$ and $S^2$ are independent. Thus, $\frac{n{S}^2}{{\sigma }^2}$ (a function of $S^2$) is independent from $Z^2$ (a function of $\bar{X}$). Now, let $U=\frac{n{S}^2}{{\sigma }^2}$ and $V=Z^2$. Since $U$ and $V$ are independent, and also we have $W=U+V$ from above derivation, the mgf $$M_W(t)=M_{U+V}(t)=M_U(t)M_V(t).$$ Since $W\sim {\chi }_n^2$ and $V\sim {\chi }_1^2$, we can further write $$(1-2t{)}^{-n/2}=M_U(t)(1-2t{)}^{-1/2},t<\frac{1}{2},$$ which implies that the mgf of $U$ is $M_U(t)=(1-2t{)}^{-(n-1)/2},t<\frac{1}{2}$, which is exactly the mgf of ${\chi }_{n-1}^2$. Hence, $U=\frac{n{S}^2}{{\sigma }^2}\sim {\chi }_{n-1}^2$.

(iii) We will use the following definition of $t$-distribution $t_k$: $\frac{Z}{\sqrt{Y/k}}\sim t_k$ where $Z\sim 𝒩(0,1)$, $Y\sim {\chi }_k^2$, and $Z$ and $Y$ are independent.

After using this definition, it is easy to prove (iii) with (ii), as follows: $$\begin{array}{cc}\frac{\bar{X}-\mu }{S/\sqrt{n-1}}& =\frac{\sqrt{n}(\bar{X}-\mu )/\sigma }{\sqrt{n}S/(\sigma \sqrt{n-1})}\\ & =\frac{\sqrt{n}(\bar{X}-\mu )/\sigma }{\sqrt{\frac{n{S}^2}{{\sigma }^2}/n-1}}.\end{array}$$ By (ii), $\frac{n{S}^2}{{\sigma }^2}\sim {\chi }_{n-1}^2$. Also, we know that $\frac{\sqrt{n}(\bar{X}-\mu )}{\sigma }$ and $\frac{n{S}^2}{{\sigma }^2}$ are independent since $\bar{X}$ and $S^2$ are independent by (i). Then, it follows by the above definition that $\frac{\bar{X}-\mu }{S/\sqrt{n-1}}\sim t_{n-1}$.

$◻$

Using this proposition, we can prove the following theorem. Again, before discussing this confidence interval, let us introduce a notation:

• $t_{\alpha ,\nu }$ is the upper percentile of $t_{\nu }$ at level $\alpha$, i.e. it satisfies $\mathbb{P}(T\ge t_{\alpha ,\nu })=\alpha$ where $T\sim t_{\nu }$.

Proof. By (iii) in the previous proposition, we have $T=\frac{\bar{X}-\mu }{S/\sqrt{n-1}}\sim t_{n-1}$. Since $t_{n-1}$ is independent from $\mu$, $T$ is a pivotal quantity of $\mu$. Hence, we set $$1-\alpha =1-\mathbb{P}(T\ge t_{\alpha /2,n-1}))-\mathbb{P}(T\le -t_{\alpha /2,n-1})=\mathbb{P}(-t_{\alpha /2,n-1}\le T\le t_{\alpha /2,n-1})$$ where $t_{\alpha /2,n-1}$ is a constant ($t$-distribution is symmetric (about $x=0$), so we have $\mathbb{P}(T\le -t_{\alpha /2,n-1})=\alpha /2$). It follows that $$\begin{array}{cc}1-\alpha & =\mathbb{P}(-t_{\alpha /2,n-1}\le T\le t_{\alpha /2,n-1})\\ & =\mathbb{P}(-t_{\alpha /2,n-1}\le \frac{\bar{X}-\mu }{S/\sqrt{n-1}}\le t_{\alpha /2,n-1})\\ & =\mathbb{P}(-t_{\alpha /2,n-1}\frac{S}{\sqrt{n-1}}\le \bar{X}-\mu \le t_{\alpha /2,n-1}\frac{S}{\sqrt{n-1}})\\ & =\mathbb{P}(t_{\alpha /2,n-1}\frac{S}{\sqrt{n-1}}\ge \mu -\bar{X}\ge -t_{\alpha /2,n-1}\frac{S}{\sqrt{n-1}})\\ & =\mathbb{P}(-t_{\alpha /2,n-1}\frac{S}{\sqrt{n-1}}\le \mu -\bar{X}\le t_{\alpha /2,n-1}\frac{S}{\sqrt{n-1}})& (\text{rewrite})\\ & =\mathbb{P}(\bar{X}-t_{\alpha /2,n-1}\frac{S}{\sqrt{n-1}}\le \mu \le \bar{X}+t_{\alpha /2,n-1}\frac{S}{\sqrt{n-1}}).\\ \end{array}$$ The result follows.

$◻$

Difference in means of two normal distributions

Sometimes, apart from estimating mean of a Template:Colored em normal distribution, we would like to estimate the Template:Colored em in means of Template:Colored em normal distributions for making comparison. For example, apart from estimating the mean amount of time (lifetime) for a bulb until it burns out, we are often interested in estimating the Template:Colored em between life of two different bulbs, so that we know which of the bulbs will last longer in average, and then we know that bulb has a higher "quality".

First, let us discuss the case where the two normal distributions are Template:Colored em.

Now, the problem is that how should we construct a confidence interval for the Template:Colored em in two means. It seems that we can just construct two $1-\alpha$ confidence intervals $[L(𝐗),U(𝐗)],[L(𝐘),U(𝐘)]$ for each of the two means ${\mu }_X,{\mu }_Y$ respectively. Then, the $1-\alpha$ confidence interval for ${\mu }_X-{\mu }_Y$ is $[L(𝐗)-L(𝐘),U(𝐗)-U(𝐘)]$. However, this is indeed incorrect since when we have $\mathbb{P}(L(𝐗)\le {\mu }_X\le U(𝐗))=1-\alpha$ and $\mathbb{P}(L(𝐘)\le {\mu }_Y\le U(𝐘))=1-\alpha$, it does Template:Colored em mean that $\mathbb{P}(L(𝐗)-L(𝐘)\le {\mu }_X-{\mu }_Y\le U(𝐗)-U(𝐘))=1-\alpha$ (there are no results in probability that justify this).

On the other hand, it seems that since $\{L(𝐗)\le {\mu }_X\le U(𝐗)\}$ and $\{L(𝐘)\le {\mu }_Y\le U(𝐘)\}$ are independent (since the normal distributions we are considering are independent), then we have $$\mathbb{P}(L(𝐗)\le {\mu }_X\le U(𝐗)\text{ and }L(𝐘)\le {\mu }_Y\le U(𝐘))=(1-\alpha {)}^2.$$ Then, when $L(𝐗)\le {\mu }_X\le U(𝐗)$ and $L(𝐘)\le {\mu }_Y\le U(𝐘)$, we have $$L(𝐗)-U(𝐘)\le {\mu }_X-{\mu }_Y\le U(𝐗)-L(𝐘),$$ so $$\mathbb{P}(L(𝐗)-U(𝐘)\le {\mu }_X-{\mu }_Y\le U(𝐗)-L(𝐘))=(1-\alpha {)}^2,$$ which means $[L(𝐗)-U(𝐘),U(𝐗)-L(𝐘)]$ is a $(1-\alpha {)}^2$ confidence interval.

However, this is actually also incorrect. The flaw is that "when $L(𝐗)\le {\mu }_X\le U(𝐗)$ and $L(𝐘)\le {\mu }_Y\le U(𝐘)$, we have $L(𝐗)-U(𝐘)\le {\mu }_X-{\mu }_Y\le U(𝐗)-L(𝐘)$" only means $$\{L(𝐗)\le {\mu }_X\le U(𝐗)\text{ and }L(𝐘)\le {\mu }_Y\le U(𝐘)\}\subseteq \{L(𝐗)-U(𝐘)\le {\mu }_X-{\mu }_Y\le U(𝐗)-L(𝐘)\}$$ (we do not have the reverse subset inclusion in general). This in turn means $$(1-\alpha {)}^2=\mathbb{P}(L(𝐗)\le {\mu }_X\le U(𝐗)\text{ and }L(𝐘)\le {\mu }_Y\le U(𝐘))\le \mathbb{P}(L(𝐗)-U(𝐘)\le {\mu }_X-{\mu }_Y\le U(𝐗)-L(𝐘)).$$ So, $[L(𝐗)-U(𝐘),U(𝐗)-L(𝐘)]$ is actually Template:Colored em a $(1-\alpha {)}^2$ confidence interval (in general).

So, the above two "methods" to construct confidence intervals for difference in means of two independent normal distributions actually do not work. Indeed, we do Template:Colored em use the confidence interval for each of the two means, which is constructed previously, to construct a confidence interval for difference in the two means. Instead, we consider a Template:Colored em of the difference in the two means, which is a standard way for constructing confidence intervals.

Template:Colored theorem Template:Colored remark Template:Colored exercise Now, we will prove the above theorem based on the result shown in the previous exercise:

Proof. Let $Z=\frac{(\bar{X}-\bar{Y})-({\mu }_X-{\mu }_Y)}{\sqrt{{\sigma }_X^2/n+{\sigma }_Y^2/m}}\sim 𝒩(0,1)$ (from the previous exercise). Then, $Z$ is a pivotal quantity of ${\mu }_X-{\mu }_Y$. Hence, we have $$\begin{array}{cc}1-\alpha & =\mathbb{P}(-z_{\alpha /2}\le Z\le z_{\alpha /2})\\ & =\mathbb{P}(-z_{\alpha /2}\le \frac{(\bar{X}-\bar{Y})-({\mu }_X-{\mu }_Y)}{\sqrt{{\sigma }_X^2/n+{\sigma }_Y^2/m}}\le z_{\alpha /2})\\ & =\mathbb{P}(-z_{\alpha /2}\sqrt{{\sigma }_X^2/n+{\sigma }_Y^2/m}\le (\bar{X}-\bar{Y})-({\mu }_X-{\mu }_Y)\le z_{\alpha /2}\sqrt{{\sigma }_X^2/n+{\sigma }_Y^2/n})\\ & =\mathbb{P}(z_{\alpha /2}\sqrt{\frac{{\sigma }_X^2}{n}+\frac{{\sigma }_Y^2}{m}}\ge ({\mu }_X-{\mu }_Y)-(\bar{X}-\bar{Y})\ge -z_{\alpha /2}\sqrt{\frac{{\sigma }_X^2}{n}+\frac{{\sigma }_Y^2}{m}})\\ & =\mathbb{P}((\bar{X}-\bar{Y})-z_{\alpha /2}\sqrt{\frac{{\sigma }_X^2}{n}+\frac{{\sigma }_Y^2}{m}}\le {\mu }_X-{\mu }_Y\le (\bar{X}-\bar{Y})+z_{\alpha /2}\sqrt{\frac{{\sigma }_X^2}{n}+\frac{{\sigma }_Y^2}{m}}).\\ \end{array}$$

$◻$

Template:Colored example Now, we will consider the case where the variances are Template:Colored em. In this case, the construction of the confidence interval for the difference in means is more complicated, and even more complicated when ${\sigma }_X^2\ne {\sigma }_Y^2$. Thus, we will only discuss the case where ${\sigma }_X^2={\sigma }_Y^2$ is unknown. As you may expect, we will also use some results mentioned previously for constructing confidence interval for $\mu$ when ${\sigma }^2$ is unknown in this case. Template:Colored theorem Template:Colored remark

Proof. Let $Z=\frac{(\bar{X}-\bar{Y})-({\mu }_X-{\mu }_Y)}{\sqrt{{\sigma }^2/n+{\sigma }^2/m}}\sim 𝒩(0,1)$ (the reason for this to follow $𝒩(0,1)$ is shown in a previous exercise). From a previous result, we know that $V=\frac{n{S}_X^2}{{\sigma }^2}\sim {\chi }_{n-1}^2$ and $W=\frac{m{S}_Y^2}{{\sigma }^2}\sim {\chi }_{m-1}^2$. Then, we know that the mgf of $V$ is $M_V(t)=(1-2t{)}^{-(n-1)/2}$ and the mgf of $W$ is $M_W(t)=(1-2t{)}^{-(m-1)/2}$. Since the distributions $𝒩({\mu }_X,{\sigma }^2)$ and $𝒩({\mu }_Y,{\sigma }^2)$ are independent, the mgf of $U=V+W$ is $$M_U(t)=M_{V+W}(t)=M_V(t)M_W(t)=(1-2t{)}^{-(n-1)/2-(m-1)/2}=(1-2t{)}^{-(n+m-2)/2}.$$ Hence, $W\sim {\chi }_{n+m-2}^2$.

By the independence of sample mean and sample variance ($\bar{X}$ and $S_X^2$ are independent, $\bar{Y}$ and $S_Y^2$ are independent), we can deduce that $Z$ and $U$ are independent. Thus, by the definition of $t$-distribution, $$\begin{array}{cc}T& =\frac{Z}{\sqrt{U/(n+m-2)}}\\ & =\frac{((\bar{X}-\bar{Y})-({\mu }_X-{\mu }_Y))/\sqrt{{\sigma }^2/n+{\sigma }^2/m}}{\sqrt{(n{S}_X^2+m{S}_Y^2)/({\sigma }^2(n+m-2))}}\\ & =\frac{((\bar{X}-\bar{Y})-({\mu }_X-{\mu }_Y))/\left(\cancel{\sigma }\sqrt{1/n+1/m}\right)}{\cancel{(1/\sigma )}\sqrt{(n{S}_X^2+m{S}_Y^2)/(n+m-2)}}& (\text{this step is not possible without the equal variance assumption})\\ & =\frac{(\bar{X}-\bar{Y})-({\mu }_X-{\mu }_Y)}{\sqrt{1/n+1/m}\sqrt{(n{S}_X^2+m{S}_Y^2)/(n+m-2)}}\\ & =((\bar{X}-\bar{Y})-({\mu }_X-{\mu }_Y))/\sqrt{\frac{n{S}_X^2+m{S}_Y^2}{n+m-2}(\frac{1}{n}+\frac{1}{m})}\\ & =\frac{(\bar{X}-\bar{Y})-({\mu }_X-{\mu }_Y)}{R}& (R=\sqrt{\frac{n{S}_X^2+m{S}_Y^2}{n+m-2}(\frac{1}{n}+\frac{1}{m})})\end{array}$$ follows $t_{n+m-2}$. Therefore, $T\sim t_{n+m-2}$ is a pivotal quantity of ${\mu }_X-{\mu }_Y$. Hence, we have $$\begin{array}{cc}1-\alpha & =\mathbb{P}(-t_{\alpha /2,n+m-2}\le T\le t_{\alpha /2,n+m-2})\\ & =\mathbb{P}(-t_{\alpha /2,n+m-2}\le \frac{(\bar{X}-\bar{Y})-({\mu }_X-{\mu }_Y)}{R}\le t_{\alpha /2,n+m-2})\\ & =\mathbb{P}(-t_{\alpha /2,n+m-2}R\le (\bar{X}-\bar{Y})-({\mu }_X-{\mu }_Y)\le t_{\alpha /2,n+m-2}R)\\ & =\mathbb{P}((\bar{X}-\bar{Y})+t_{\alpha /2,n+m-2}R\ge ({\mu }_X-{\mu }_Y)\ge (\bar{X}-\bar{Y})-t_{\alpha /2,n+m-2}R)\\ & =\mathbb{P}((\bar{X}-\bar{Y})-t_{\alpha /2,n+m-2}R\le ({\mu }_X-{\mu }_Y)\le (\bar{X}-\bar{Y})+t_{\alpha /2,n+m-2}R)\\ & =\mathbb{P}((\bar{X}-\bar{Y})-t_{\alpha /2,n+m-2}\sqrt{\frac{n{S}_X^2+m{S}_Y^2}{n+m-2}(\frac{1}{n}+\frac{1}{m})}\le ({\mu }_X-{\mu }_Y)\le (\bar{X}-\bar{Y})+t_{\alpha /2,n+m-2}\sqrt{\frac{n{S}_X^2+m{S}_Y^2}{n+m-2}(\frac{1}{n}+\frac{1}{m})}).\\ \end{array}$$ The result follows.

$◻$

Now, what if the two normal distributions concerned are Template:Colored em? Clearly, we cannot use the above results anymore, and we need to develop a new method to construct a confidence interval for the difference of means in this case. In this case, we need to consider the notion of Template:Colored em.

Proof.

1. Independence of $D_1,\dots ,D_n$:

Since $X_1,\dots ,X_n$ are independent, and $Y_1,\dots ,Y_n$ are independent, it follows that $X_1-Y_1,\dots ,X_n-Y_n$ are independent, which is what we want to show.

2. $D_1,\dots ,D_n\sim 𝒩({\mu }_X-{\mu }_Y,{\sigma }_X^2+{\sigma }_Y^2+2\mathrm{Cov}(X,Y))$:

• To show that $D_1,\dots ,D_n$ still follow normal distribution, we can consider the pdf of $D_i$ for each $i\in \{1,\dots ,n\}$. The pdf can be obtained using the transformation of random variables formula: e.g., let $U=X-Y$ and $V=Y$ where $X\sim 𝒩({\mu }_X,{\sigma }_X^2)$ and $𝒩({\mu }_Y,{\sigma }_Y^2)$. Then, the pdf of $U$ obtained, which is the pdf of $D_i$, should be in the form of normal distribution.

• However, since the actual derivation process is somewhat complicated, it is omitted here.

• Of course, the mean and variance of $D_i$ can be observed from the pdf of $D_i$ determined previously. Alternatively, before determining the pdf of $D_i$, we can also know that the mean of $D_i$ is $𝔼[D_i]=𝔼[X_i]-𝔼[Y_i]={\mu }_X-{\mu }_Y$ (we use the linearity of expectation here, which does not require independence assumption), and the variance of $D_i$ is $\mathrm{Var}(D_i)=\mathrm{Var}(X_i)+(-1{)}^2\mathrm{Var}(Y_i)+2\mathrm{Cov}(X_i,-Y_i)={\sigma }_X^2+{\sigma }_Y^2+2(-1)\mathrm{Cov}(X_i,Y_i)={\sigma }_X^2+{\sigma }_Y^2-2\mathrm{Cov}(X,Y)$ ($X\sim 𝒩({\mu }_X,{\sigma }_X^2)$ and $Y\sim 𝒩({\mu }_Y,{\sigma }_Y^2)$).

$◻$

Proof. From the previous proposition, we know that $D_1,\dots ,D_n$ is a random sample from $𝒩({\mu }_D,{\sigma }_D^2)$. Since ${\sigma }_D^2$ is known, it follows from a previous theorem that a $1-\alpha$ confidence interval for ${\mu }_D={\mu }_X-{\mu }_Y$ is $$[\bar{D}-z_{\alpha /2}\frac{{\sigma }_D}{\sqrt{n}},\bar{D}+z_{\alpha /2}\frac{{\sigma }_D}{\sqrt{n}}].$$

$◻$

Confidence intervals for variances of normal distributions

Variance of a normal distribution

After discussing the confidence intervals for means of normal distributions, let us consider the confidence intervals for Template:Colored em of normal distributions. Similarly, we need to consider a pivotal quantity of ${\sigma }^2$. Can you suggest a pivotal quantity of ${\sigma }^2$, based on a previous result discussed?

Recall that we have $\frac{n{S}^2}{{\sigma }^2}\sim {\chi }_{n-1}^2$, and ${\chi }_{n-1}^2$ is independent from ${\sigma }^2$ with some suitable assumptions. Thus, this result gives us a pivotal quantity of ${\sigma }^2$, namely $\frac{n{S}^2}{{\sigma }^2}$. Before discussing the theorem for constructing a confidence interval for ${\sigma }^2$. Let us introduce a notation:

• ${\chi }_{\alpha ,\nu }^2$ is the upper percentile of ${\chi }_{\nu }^2$ at level $\alpha$, i.e. it satisfies $\mathbb{P}(X\ge {\chi }_{\alpha ,\nu }^2)=\alpha$ where $X\sim {\chi }_{\nu }^2$.

Some values of ${\chi }_{\alpha ,\nu }^2$ can be found in the chi-squared table.

• To find the value of ${\chi }_{\alpha ,\nu }^2$, locate the row for $\nu$ degrees of freedom and the column for "probability content" $\alpha$.

Proof. Since $Y=\frac{n{S}^2}{{\sigma }^2}\sim {\chi }_{n-1}^2$, set $$1-\alpha =(1-\alpha /2)-\alpha /2=\mathbb{P}(Y\ge {\chi }_{1-\alpha /2,n-1}^2)-\mathbb{P}(Y\ge {\chi }_{\alpha /2,n-1}^2)=\mathbb{P}({\chi }_{1-\alpha /2,n-1}^2\le Y<{\chi }_{\alpha /2,n-1}^2)=\mathbb{P}({\chi }_{1-\alpha /2,n-1}^2\le Y<{\chi }_{\alpha /2,n-1}^2).$$ Then, we have $$\begin{array}{cc}1-\alpha & =\mathbb{P}({\chi }_{1-\alpha /2,n-1}^2\le \frac{n{S}^2}{{\sigma }^2}\le {\chi }_{\alpha /2,n-1}^2)\\ & =\mathbb{P}(\frac{{\chi }_{1-\alpha /2,n-1}^2}{n{S}^2}\le \frac{1}{{\sigma }^2}\le \frac{{\chi }_{\alpha /2,n-1}^2}{n{S}^2})\\ & =\mathbb{P}(\frac{n{S}^2}{{\chi }_{1-\alpha /2,n-1}^2}\ge {\sigma }^2\ge \frac{n{S}^2}{{\chi }_{\alpha /2,n-1}^2})\\ & =\mathbb{P}(\frac{n{S}^2}{{\chi }_{\alpha /2,n-1}^2}\le {\sigma }^2\le \frac{n{S}^2}{{\chi }_{1-\alpha /2,n-1}^2}).\\ \end{array}$$ The result follows.

$◻$

Ratio of variances of two independent normal distributions

Similar to the case for means, we would also sometimes like to compare the variances of two normal distributions. One may naturally expect that we should construct a confidence interval for Template:Colored em in variances, similar to the case for means. However, there are simple ways to do this, since we do not have some results that help with this construction. Therefore, we need to consider an alternative way to Template:Colored em the variances, without using the Template:Colored em in variances. Can you suggest a way?

Recall the definition of Template:Colored em in point estimation. Template:Colored em gives us a nice way to compare two variances without considering their difference, where the Template:Colored em of two variances is considered. Fortunately, we have some results that help us to construct a confidence interval for the Template:Colored em of two variances.

Recall that the definition of $F$-distribution: if $U\sim {\chi }_{{\nu }_1}^2$ and $V\sim {\chi }_{{\nu }_2}^2$ are independent, then $\frac{U/{\nu }_1}{V/{\nu }_2}$ follows the $F$-distribution with ${\nu }_1$ and ${\nu }_2$ degrees of freedom, denoted by $F_{{\nu }_1,{\nu }_2}$. From the definition of $F$-distribution, we can see that it involves a ratio of two independent chi-squared random variables. How can it be linked to the ratio of two variances?

Recall that we have $\frac{n{S}^2}{{\sigma }^2}\sim {\chi }_{n-1}^2$ with some suitable assumptions. This connects the variance with the chi-squared random variable, and thus we can use this property together with the definition of $F$-distribution to construct a pivotal quantity, and hence a confidence interval.

Let us introduce a notation before discussing the construction of confidence interval:

• $F_{\alpha ,{\nu }_1,{\nu }_2}$ is the upper percentile of $F_{{\nu }_1,{\nu }_2}$ at level $\alpha$, i.e. it satisfies $\mathbb{P}(X\ge F_{\alpha ,{\nu }_1,{\nu }_2})=\alpha$.

Some values of $F_{\alpha ,{\nu }_1,{\nu }_2}$ can be found in $F$-tables (there is different $F$-tables for different values of $\alpha$, and the row and column of each table indicates the first and second degrees of freedom respectively). Also, using the property that $F_{\alpha ,{\nu }_1,{\nu }_2}=\frac{1}{F_{1-\alpha ,{\nu }_2,{\nu }_1}}$, we can obtain some more values of $F_{\alpha ,{\nu }_1,{\nu }_2}$ which are not included in the $F$-tables. Template:Colored theorem Template:Colored remark

Proof. By the assumptions, we have $$\frac{n{S}_X^2}{{\sigma }_X^2}\sim {\chi }_{n-1}^2\text{ and }\frac{m{S}_Y^2}{{\sigma }_Y^2}\sim {\chi }_{m-1}^2.$$ Thus, by the definition of $F$-distribution, we have $$\frac{m{S}_Y^2}{{\sigma }_Y^2(m-1)}/\frac{n{S}_X^2}{{\sigma }_X^2(n-1)}=\frac{m(n-1)S_Y^2{\sigma }_X^2}{n(m-1)S_X^2{\sigma }_Y^2}\sim F_{m-1,n-1},$$ which is a pivotal quantity of ${\sigma }_X^2/{\sigma }_Y^2$. Hence, we have $$\begin{array}{cc}1-\alpha & =(1-\alpha /2)-\alpha /2\\ & =\mathbb{P}(F_{1-\alpha /2,m-1,n-1}\le \frac{m(n-1)S_Y^2{\sigma }_X^2}{n(m-1)S_Y^2{\sigma }_X^2}\le F_{\alpha /2,m-1,n-1})\\ & =\mathbb{P}(\frac{n(m-1)S_X^2}{m(n-1)S_Y^2}\cdot F_{1-\alpha /2,m-1,n-1}\le \frac{{\sigma }_X^2}{{\sigma }_Y^2}\le \frac{n(m-1)S_X^2}{m(n-1)S_Y^2}\cdot F_{\alpha /2,m-1,n-1}),\\ \end{array}$$ as desired.

$◻$

Apart from using this confidence interval to Template:Colored em variances (or standard deviations), it can also be useful to Template:Colored em some assumptions about variances. Let us illustrate these two usages in the following examples. Template:Colored example Template:Colored example Template:Colored remark

Approximated confidence intervals for means

Previously, the distributions for the population are assumed to be normal, but the distributions are often Template:Colored em normal in reality. So, does it mean our previous discussions are meaningless in reality? No. The discussions are indeed still quite meaningful in reality, since we can use the Template:Colored em to "connect" the distributions in reality (which are usually not normal) to normal distribution. Through this, we can construct Template:Colored em confidence intervals, since we use central limit theorem for approximation.

To be more precise, recall that the Template:Colored em suggests that $\frac{\bar{X}-\mu }{\sigma /\sqrt{n}}\stackrel{d}{\to }Z\sim 𝒩(0,1)$ with some suitable assumptions. Therefore, if the sample size $n$ is large enough (a rule of thumb: at least 30), then $\frac{\bar{X}-\mu }{\sigma /\sqrt{n}}$ follows Template:Colored em standard normal distribution. Hence, $\frac{\bar{X}-\mu }{\sigma /\sqrt{n}}$ is a pivotal quantity (approximately). Recall from the property of normal distribution that if $X_1,\dots ,X_n$ is a random sample from $𝒩(\mu ,{\sigma }^2)$, then we have $\frac{\bar{X}-\mu }{\sigma /\sqrt{n}}\sim 𝒩(0,1)$ Template:Colored em (not approximately), and we have used this for the pivotal quantity for the confidence interval for mean when variance is known, and also the confidence interval for ${\mu }_X-{\mu }_Y$ when ${\sigma }_D^2$ is known. Therefore, we can just use basically the same confidence interval in these cases, but we need to notice that such confidence intervals are approximated, but not exact since we have used the central limit theorem for constructing the pivot quantity.

Now, how about the other confidence intervals where the pivotal quantity is "not in this form"? In the confidence interval for difference in means when variance is unknown, the pivotal quantity is similar in some sense: $\frac{(\bar{X}-\bar{Y})-({\mu }_X-{\mu }_Y)}{\sqrt{{\sigma }_X^2/n+{\sigma }_Y^2/m}}\sim 𝒩(0,1)$ (see the corresponding theorem for the meaning of the notations involved). Can we use the central limit theorem to conclude that when the distributions involved are not normal (but are still independent), and the sample sizes $n$ and $m$ are both large enough, then $\frac{(\bar{X}-\bar{Y})-({\mu }_X-{\mu }_Y)}{\sqrt{{\sigma }_X^2/n+{\sigma }_Y^2/m}}\sim 𝒩(0,1)$ Template:Colored em? The answer is yes. For the proof, see the following exercise. Template:Colored exercise As a result, we know that we can again just use basically the same confidence interval in this case, but of course such confidence interval is approximated.

There are still some confidence intervals that are not considered yet. Let us first consider the confidence interval for mean when the variance is Template:Colored em.

Recall that we have mentioned that we can simply replace the "$\sigma$" by "$S$" according to the weak law of large number, which is quite intuitive. But why can we do this? Consider the following theorem. Template:Colored theorem Template:Colored remark

Proof. Under the assumption that the random sample has finite mean and variance, applying weak law of large number gives $S\stackrel{p}{\to }\sigma$ (we have shown that $S^2\stackrel{p}{\to }{\sigma }^2$, then we can just apply continuous mapping theorem to get this). Hence, $\frac{\sigma }{S}\stackrel{p}{\to }\frac{\sigma }{\sigma }=1$ ($\sigma >0$) by property of convergence in probability.

By central limit theorem, we have $\frac{\bar{X}-\mu }{\sigma /\sqrt{n}}\stackrel{d}{\to }Z\sim 𝒩(0,1)$. Thus, $$\frac{\bar{X}-\mu }{S/\sqrt{n}}=\frac{\bar{X}-\mu }{\sigma /\sqrt{n}}\cdot \frac{\sigma }{S}\stackrel{d}{\to }Z\sim 𝒩(0,1)$$ by Slutsky's theorem.

Therefore, $\frac{\bar{X}-\mu }{S/\sqrt{n}}$ is a pivotal quantity, which follows $𝒩(0,1)$ approximately. Notice that its approximated distribution, $𝒩(0,1)$, is the same as that of pivotal quantity for confidence interval for $\mu$ when ${\sigma }^2$ is known, namely $\frac{\bar{X}-\mu }{\sigma /\sqrt{n}}$. As a result, we can use similar steps to obtain the approximated confidence interval, where "$\sigma$" is replaced by "$S$".

$◻$

So far, we have not discussed how to construct an approximated confidence interval for ${\mu }_X-{\mu }_Y$ when ${\sigma }_X^2={\sigma }_Y^2={\sigma }^2$ is unknown, as well as approximated confidence intervals of variances. Since the pivotal quantities used are constructed according to some results that are exclusive to normal distributions, they all do Template:Colored em work when the distributions involved are not normal. Therefore, there are no simple ways to perform such constructions.

The following table summarizes the approximated $1-\alpha$ confidence intervals in different cases: $$\begin{array}{ccc}(\text{approximated }(1-\alpha )\text{ confidence intervals})& \text{mean}& \text{difference in means}\\ \text{known variance}& [\bar{X}-z_{\alpha /2}\frac{\sigma }{\sqrt{n}},\bar{X}+z_{\alpha /2}\frac{\sigma }{\sqrt{n}}]& [(\bar{X}-\bar{Y})-z_{\alpha /2}\sqrt{\frac{{\sigma }_X^2}{n}+\frac{{\sigma }_Y^2}{m}},(\bar{X}-\bar{Y})+z_{\alpha /2}\sqrt{\frac{{\sigma }_X^2}{n}+\frac{{\sigma }_Y^2}{m}}]\text{ OR }[\bar{D}-z_{\alpha /2}\frac{{\sigma }_D}{\sqrt{n}},\bar{D}+z_{\alpha /2}\frac{{\sigma }_D}{\sqrt{n}}](\text{paired samples})\\ \text{unknown variance}& [\bar{X}-z_{\alpha /2}\frac{S}{\sqrt{n}},\bar{X}+z_{\alpha /2}\frac{S}{\sqrt{n}}]& [\bar{D}-z_{\alpha /2}\frac{S_D}{\sqrt{n}},\bar{D}+z_{\alpha /2}\frac{S_D}{\sqrt{n}}](\text{paired samples})\end{array}$$ Template:Colored remark Template:Colored example Let us consider an application of the approximated confidence intervals. Template:Colored proposition Template:Colored remark

Proof. Since $\xi =\mathrm{𝟏}\{X\in S\}$, by the fundamental bridge between probability and expectation, we have $$𝔼[\xi ]=𝔼[\mathrm{𝟏}\{X\in S\}]=\mathbb{P}(X\in S)=p.$$

Applying the result for constructing an approximated confidence interval for mean when variance is unknown (the variance of $\xi$ is $p(1-p)$ since $\xi$ follows the Bernoulli distribution $\mathrm{Ber}(p)$ actually), an approximated $1-\alpha$ confidence interval for $p$ is $$[\bar{\xi }-z_{\alpha /2}\frac{S_{\xi }}{\sqrt{n}},\bar{\xi }+z_{\alpha /2}\frac{S_{\xi }}{\sqrt{n}}].$$

$◻$