Statistics/Hypothesis Testing

Introduction

In previous chapters, we have discussed two methods for Template:Colored em, namely Template:Colored em and Template:Colored em. Estimating unknown parameters is an important area in statistical inference, and in this chapter we will discuss another important area, namely Template:Colored em, which is related to Template:Colored em. Indeed, the concepts of Template:Colored em and Template:Colored em are closely related, as we will demonstrate.

Basic concepts and terminologies

Before discussing how to Template:Colored em hypothesis testing, and Template:Colored em the "goodness" of a hypothesis test, let us introduce some basic concepts and terminologies related to hypothesis testing first. Template:Colored definition There are two terms that classify hypotheses: Template:Colored definition Sometimes, it is not immediately clear that whether a hypothesis is simple or composite. To understand the classification of hypotheses more clearly, let us consider the following example. Template:Colored example In hypothesis tests, we consider two hypotheses: Template:Colored definition Template:Colored remark A general form of $H_0$ and $H_1$ is $H_0:\theta \in {\mathrm{\Theta }}_0$ and $H_1:\theta \in {\mathrm{\Theta }}_1$ where ${\mathrm{\Theta }}_1={\mathrm{\Theta }}_0^c$, which is the Template:Colored em of ${\mathrm{\Theta }}_0$ (with respect to $\mathrm{\Theta }$), i.e., ${\mathrm{\Theta }}_0^c=\mathrm{\Theta }\setminus {\mathrm{\Theta }}_0$ ($\mathrm{\Theta }$ is the parameter space, containing all possible values of $\theta$). The reason for choosing the complement of ${\mathrm{\Theta }}_0$ in $H_1$ is that $H_1$ is the complementary hypothesis to $H_0$, as suggested in the above definition. Template:Colored remark Template:Colored example We have mentioned that exactly one of $H_0$ and $H_1$ is assumed to be true. To make a decision, we need to Template:Colored em which hypothesis should be regarded as true. Of course, as one may expect, this decision is not perfect, and we will have some errors involved in our decision. So, we cannot say we "prove that" a particular hypothesis is true (that is, we cannot be Template:Colored em that a particular hypothesis is true). Despite this, we may "regard" (or "accept") a particular hypothesis as true (but Template:Colored em prove it as true) when we have Template:Colored em that lead us to make this decision (ideally, with small errors ). Template:Colored remark Now, we are facing with two questions. First, what evidences should we consider? Second, what is meant by "sufficient"? For the first question, a natural answer is that we should consider the observed Template:Colored em, right? This is because we are making hypothesis about the population, and the samples are taken from, and thus closely related to the population, which should help us make the decision.

To answer the second question, we need the concepts in Template:Colored em. In particular, in hypothesis testing, we will construct a so-called Template:Colored em or Template:Colored em to help us determining that Template:Colored em we should reject the Template:Colored em hypothesis (i.e., regard $H_0$ as false), and hence (naturally) regard $H_1$ as true ("accept" $H_1$) (we have assumed that exactly one of $H_0$ and $H_1$ is true, so when we regard one of them as false, we should regard another of them as true). In particular, when we do Template:Colored em reject $H_0$, we will act as if, or accept $H_0$ as true (and thus should also reject $H_1$ since exactly one of $H_0$ of $H_1$ is true).

Let us formally define the terms related to hypothesis testing in the following. Template:Colored definition Template:Colored remark Template:Colored definition Template:Colored remark Typically, we use Template:Colored em (a statistic for conducting a hypothesis test) to specify the rejection region. For instance, if the random sample is $X_1,\dots ,X_n$ and the test statistic is $\bar{X}$, the rejection region may be, say, $R=\{𝐱:\bar{x}<2\}$ (where $x_1,\dots ,x_n$ and $\bar{x}$ is observed value of $X_1,\dots ,X_n$ and $\bar{X}$ respectively). Through this, we can directly construct a hypothesis test: when $𝐱\in R$, we reject $H_0$ and accept $H_1$. Otherwise, if $𝐱\in R^c$, we accept $H_0$. So, in general, to specify the rule in a hypothesis test, we just need a Template:Colored em. After that, we will apply the test on testing $H_0$ against $H_1$. There are some terminologies related to the hypothesis tests constructed in this way: Template:Colored definition Template:Colored remark Template:Colored example As we have mentioned, the decisions made by hypothesis test should not be perfect, and errors occur. Indeed, when we think carefully, there are actually Template:Colored em of errors, as follows: Template:Colored definition We can illustrate these two types of errors more clearly using the following table.

Type I and II errors

	Accept $H_0$	Reject $H_0$
$H_0$ is true	Correct decision	Type I error
$H_0$ is false	Type II error	Correct decision

We can express $H_0:\theta \in {\mathrm{\Theta }}_0$ and $H_1:\theta \in {\mathrm{\Theta }}_0^c$. Also, assume the rejection region is $R=R(𝐗)$ (i.e., the rejection region with "$x$" replaced by "$X$"). In general, when "$R$" is put together with "$X$", we assume $R=R(𝐗)$.

Then we have some notations and expressions for Template:Colored em of making type I and II errors: (let $X_1,\dots ,X_n$ be a random sample and $𝐗=(X_1,\dots ,X_n)$)

• The probability of making a type I error, denoted by $\alpha (\theta )$, is ${\mathbb{P}}_{\theta }(𝐗\in R)$ if $\theta \in {\mathrm{\Theta }}_0$.
• The probability of making a type II error, denoted by $\beta (\theta )$, is ${\mathbb{P}}_{\theta }(𝐗\in R^c)=1-{\mathbb{P}}_{\theta }(𝐗\in R)$ if $\theta \in {\mathrm{\Theta }}_0^c$.

Template:Colored remark Notice that we have a common expression in both $\alpha (\theta )$ and $\beta (\theta )$, which is "${\mathbb{P}}_{\theta }((X_1,\dots ,X_n)\in R)$". Indeed, we can also write this expression as $${\mathbb{P}}_{\theta }((X_1,\dots ,X_n)\in R)=\{\begin{array}{cc}\alpha (\theta ),& \theta \in {\mathrm{\Theta }}_0;\\ 1-\beta (\theta ),& \theta \in {\mathrm{\Theta }}_0^c.\end{array}$$ Through this, we can observe that this expression contains all informations about the probabilities of making errors, given a hypothesis test with rejection $R$. Hence, we will give a special name to it: Template:Colored definition Template:Colored remark Template:Colored example Template:Colored example Ideally, we want to make both $\alpha (\theta )$ and $\beta (\theta )$ arbitrarily small. But this is generally impossible. To understand this, we can consider the following extreme examples:

• Set the rejection region $R$ to be $S=\{𝐱\}$, which is the set of all possible observations of random samples. Then, $\pi (\theta )=1$ for each $\theta \in \mathrm{\Theta }$. From this, of course we have $\beta (\theta )=0$, which is nice. But the serious problem is that $\alpha (\theta )=1$ due to the mindless rejection.
• Another extreme is setting the rejection region $R$ to be the empty set $\varnothing$. Then, $\pi (\theta )=0$ for each $\theta \in \mathrm{\Theta }$. From this, we have $\alpha (\theta )=0$, which is nice. But, again the serious problem is that $\beta (\theta )=1$ due to the mindless acceptance.

We can observe that to make $\alpha (\theta )$ ($\beta (\theta )$) to be very small, it is inevitable that $\beta (\theta )$ ($\alpha (\theta )$) will increase consequently, due to accepting (rejecting) "too much". As a result, we can only try to minimize the probability of making one type of error, holding the probability of making another type of error Template:Colored em.

Now, we are interested in knowing that Template:Colored em type of errors should be controlled. To motivate the choice, we can again consider the analogy of legal principle of presumption of innocence. In this case, type I error means proving an innocent guilty, and type II error means acquitting a guilty person. Then, as suggested by Blackstone's ratio, type I error is more serious and important than type II error. This motivates us to control the probability of type I error, i.e., $\alpha (\theta )$, at a specified small value ${\alpha }^{*}$, so that we can control the probability of making this more serious error. After that, we consider the tests that "control the type I error probability at this level", and the one with the smallest $\beta (\theta )$ is the "best" one (in the sense of probability of making errors).

To describe "control the type I error probability at this level" in a more precise way, let us define the following term. Template:Colored definition Template:Colored remark So, using this definition, controlling the type I error probability at a particular level $\alpha$ means that the size of the test should not exceed $\alpha$, i.e., $\underset{\theta \in {\mathrm{\Theta }}_0}{\sup }\pi (\theta )\le \alpha$ (in some other places, such test is called a Template:Colored em. Template:Colored example For now, we have focused on using Template:Colored em to conduct hypothesis tests. But this is not the only way. Alternatively, we can make use of $p$-value. Template:Colored definition Template:Colored remark The following theorem allows us to use $p$-value for hypothesis testing. Template:Colored theorem

Proof. (Partial) We can prove "if" and "only if" directions at once. Let us first consider the case 1 in the definition of $p$-value. By definitions, $p$-value is $\underset{\theta \in {\mathrm{\Theta }}_0}{\sup }{\mathbb{P}}_{\theta }(T(𝐗)\le T^{*}(𝐱))$ and $\alpha =\underset{\theta \in {\mathrm{\Theta }}_0}{\sup }\pi (\theta )=\underset{\theta \in {\mathrm{\Theta }}_0}{\sup }{\mathbb{P}}_{\theta }(T(𝐗)\le T^{*}(𝐱))$ (Define $T^{*}(𝐗)$ such that $T(𝐗)\le T^{*}(𝐱)⟺(X_1,\dots ,X_n)\in R$.). Then, we have $$\begin{array}{cc}p\text{-value}\le \alpha & ⟺\underset{\theta \in {\mathrm{\Theta }}_0}{\sup }{\mathbb{P}}_{\theta }(T(𝐗)\le T(𝐱))\le \underset{\theta \in {\mathrm{\Theta }}_0}{\sup }{\mathbb{P}}_{\theta }(T(𝐗)\le T^{*}(𝐱))\\ & ⟺T(𝐱)\le T^{*}(𝐱)& (\text{by some omitted arguments and the monotonicity of cdf})\\ & ⟺(x_1,\dots ,x_n)\in \{(y_1,\dots ,y_n):T(y_1,\dots ,y_n)\le T^{*}(𝐱)\}& (x_1,\dots ,x_n\text{ are realizations of }{X}_1,\dots ,X_n\text{ respectively})\\ & ⟺(x_1,\dots ,x_n)\in R& (\text{defined above})\\ & ⟺H_0\text{ is rejected at significance level }\alpha .& (\text{the test with power function }\pi (\theta )\text{ is size }\alpha \text{ test})\end{array}$$ For other cases, the idea is similar (just the directions of inequalities for $T$ are different).

$◻$

Evaluating a hypothesis test

After discussing some basic concepts and terminologies, let us now study some ways to evaluate goodness of a hypothesis test. As we have previously mentioned, we want the probability of making type I errors and type II errors to be small, but we have mentioned that it is generally impossible to make both probabilities to be arbitrarily small. Hence, we have suggested to control the type I error, using the size of a test, and the "best" test should the one with the smallest probability of making type II error, after controlling the type I error.

These ideas lead us to the following definitions. Template:Colored definition Using this definition, instead of saying "best" test (test with the smallest type II error probability), we can say "a test with the most power", or in other words, the "most powerful test". Template:Colored definition Template:Colored remark

Constructing a hypothesis test

There are many ways of constructing a hypothesis test, but of course not all are good (i.e., "powerful"). In the following, we will provide some common approaches to construct hypothesis tests. In particular, the following lemma is very useful for constructing a MP test with size $\alpha$.

Neyman-Pearson lemma

Proof. Let us first consider the case where the underlying distribution is continuous. Template:Color, the "size" requirement for being a UMP test is satisfied immediately. So, it suffices to show that $\phi$ satisfies the "UMP" requirement for being a MP test.

Notice that in this case, "${\mathrm{\Theta }}_1$" is simply $\{{\theta }_1\}$. So, for every test $\psi$ with rejection region $R^{*}\ne R$ and ${\pi }_{\psi }({\theta }_0)\le \alpha$, we will proceed to show that ${\pi }_{\phi }({\theta }_1)\ge {\pi }_{\psi }({\theta }_1)$.

Since $$\begin{array}{cc}{\pi }_{\phi }({\theta }_1)-{\pi }_{\psi }({\theta }_1)& ={\mathbb{P}}_{{\theta }_1}((X_1,\dots ,X_n)\in R)-{\mathbb{P}}_{{\theta }_1}((X_1,\dots ,X_n)\in R^{*})\\ & =\int \cdots \underset{R}{\overset{}{\int }}ℒ({\theta }_1;𝐱)d{x}_n\cdots d{x}_1-\int \cdots \underset{R^{*}}{\overset{}{\int }}ℒ({\theta }_1;𝐱)d{x}_n\cdots d{x}_1\\ & =\int \cdots \underset{R}{\overset{}{\int }}ℒ({\theta }_1;𝐱)d{x}_n\cdots d{x}_1-\int \cdots \underset{R\cap R^{*}}{\overset{}{\int }}ℒ({\theta }_1;𝐱)d{x}_n\cdots d{x}_1-\left(\int \cdots \underset{R^{*}}{\overset{}{\int }}ℒ({\theta }_1;𝐱)d{x}_n\cdots d{x}_1-\int \cdots \underset{R\cap R^{*}}{\overset{}{\int }}ℒ({\theta }_1;𝐱)d{x}_n\cdots d{x}_1\right)\\ & =\int \cdots \underset{R\setminus R^{*}}{\overset{}{\int }}ℒ({\theta }_1;𝐱)d{x}_n\cdots d{x}_1-\int \cdots \underset{R^{*}\setminus R}{\overset{}{\int }}ℒ({\theta }_1;𝐱)d{x}_n\cdots d{x}_1\\ & \ge \frac{1}{k}\int \cdots \underset{R\setminus R^{*}}{\overset{}{\int }}ℒ({\theta }_0;𝐱)d{x}_n\cdots d{x}_1-\frac{1}{k}\int \cdots \underset{R^{*}\setminus R}{\overset{}{\int }}ℒ({\theta }_0;𝐱)d{x}_n\cdots d{x}_1(\text{In }R,ℒ({\theta }_1;𝐱)\ge \frac{1}{k}ℒ({\theta }_0;𝐱).\text{ In }{R}^c,ℒ({\theta }_1;𝐱)<\frac{1}{k}ℒ({\theta }_0;𝐱)⟺-ℒ({\theta }_1;𝐱)>-\frac{1}{k}ℒ({\theta }_0;𝐱))\\ & =\frac{1}{k}\int \cdots \underset{R\setminus R^{*}}{\overset{}{\int }}ℒ({\theta }_0;𝐱)d{x}_n\cdots d{x}_1+\frac{1}{k}\int \cdots \underset{R\cap R^{*}}{\overset{}{\int }}ℒ({\theta }_0;𝐱)d{x}_n\cdots d{x}_1-(\frac{1}{k}\int \cdots \underset{R^{*}\setminus R}{\overset{}{\int }}ℒ({\theta }_0;𝐱)d{x}_n\cdots d{x}_1+\frac{1}{k}\int \cdots \underset{R\cap R^{*}}{\overset{}{\int }}ℒ({\theta }_0;𝐱)d{x}_n\cdots d{x}_1)\\ & =\frac{1}{k}\int \cdots \underset{R}{\overset{}{\int }}ℒ({\theta }_0;𝐱)d{x}_n\cdots d{x}_1-\frac{1}{k}\int \cdots \underset{R^{*}}{\overset{}{\int }}ℒ({\theta }_0;𝐱)d{x}_n\cdots d{x}_1\\ & =\frac{1}{k}(\underset{=\alpha }{\underbrace{{\mathbb{P}}_{{\theta }_0}((X_1,\dots ,X_n)\in R)}}-\underset{\le \alpha }{\underbrace{{\mathbb{P}}_{{\theta }_0}((X_1,\dots ,X_n)\in R^{*})}})\\ & \ge \frac{1}{k}(\alpha -\alpha )=0,\end{array}$$ we have ${\pi }_{\varphi }({\theta }_1)\ge {\pi }_{\psi }({\theta }_1)$ as desired.

For the case where the underlying distribution is discrete, the proof is very similar (just replace the integrals with sums), and hence omitted.

$◻$

Template:Colored remark Even if the hypotheses involved in the Neyman-Pearson lemma are simple, with some conditions, we can use the lemma to construct a UMP test for testing Template:Colored em null hypothesis against Template:Colored em alternative hypothesis. The details are as follows: For testing $H_0:\theta \le {\theta }_0\text{vs.}{H}_1:\theta >{\theta }_0$

1. Find a MP test $\phi$ with size $\alpha$, for testing $H_0:\theta ={\theta }_0\text{vs.}{H}_1:\theta ={\theta }_1>{\theta }_0$ using the Neyman-Pearson lemma, where ${\theta }_1$ is an arbitrary value such that ${\theta }_1>{\theta }_0$.
2. Template:Colored em, then the test $\phi$ has the greatest power for Template:Colored em $\theta \in {\mathrm{\Theta }}_1=\{\vartheta :\vartheta >{\theta }_0\}$. So, the test $\phi$ is a UMP test with size $\alpha$ for testing $H_0:\theta ={\theta }_0\text{vs.}{H}_1:\theta >{\theta }_0$
3. Template:Colored em, then it means that the size of the test $\phi$ is still $\alpha$, even if the null hypothesis is changed to $H_0:\theta \le \theta$. So, after changing $H_0:\theta ={\theta }_0$ to $H_0:\theta \le {\theta }_0$ and not changing $H_1$ (also adjusting the parameter space) for the test $\phi$, the test $\phi$ still satisfies the "MP" requirement (because of not changing $H_1$, so the result in step 2 still applies), and also the test $\phi$ will satisfy the "size" requirement (because of changing $H_0$ in this way). Hence, the test $\phi$ is a UMP test with size $\alpha$ for testing $H_0:\theta \le {\theta }_0\text{vs.}{H}_1:\theta >{\theta }_0$.

For testing $H_0:\theta \ge {\theta }_0\text{vs.}{H}_1:\theta <{\theta }_0$, the steps are similar. But in general, there is no UMP test for testing $H_0:\theta ={\theta }_0\text{vs.}{H}_1:\theta \ne {\theta }_0$.

Of course, when the condition in step 3 holds but that in step 2 does not hold, the test $\phi$ in step 1 is a UMP test with size $\alpha$ for testing $H_0:\theta \le {\theta }_0\text{vs.}{H}_1:\theta ={\theta }_1$ where ${\theta }_1$ is a constant (which is larger than ${\theta }_0$, or else $H_1$ and $H_0$ are not disjoint). However, the hypotheses are generally not in this form. Template:Colored example Template:Colored remark Now, let us consider another example where the underlying distribution is discrete. Template:Colored example

Likelihood-ratio test

Previously, we have suggested using the Neyman-Pearson lemma to construct MPT for testing simple null hypothesis against simple alternative hypothesis. However, when the hypotheses are composite, we may not be able to use the Neyman-Pearson lemma. So, in the following, we will give a general method for constructing tests for any hypotheses, not limited to simple hypotheses. But we should notice that the tests constructed are not necessarily UMPT. Template:Colored definition Template:Colored remark

Relationship between hypothesis testing and confidence intervals

We have mentioned that there are similarities between hypothesis testing and confidence intervals. In this section, we will introduce a theorem suggesting how to construct a hypothesis test from a confidence interval (or in general, confidence Template:Colored em), and vice versa. Template:Colored theorem

Proof. For the first part, since $R({\theta }_0)$ is the rejection region of the size $\alpha$ test, we have $${\mathbb{P}}_{{\theta }_0}(𝐗\in R)=\alpha .$$ Hence, the coverage probability for the random set $C(𝐗)$ is $${\mathbb{P}}_{{\theta }_0}({\theta }_0\in C(𝐗))={\mathbb{P}}_{{\theta }_0}(𝐗\in R^c)=1-\alpha ,$$ which means that the random set $C(𝐗)$ is a $1-\alpha$ confidence set of ${\theta }_0$.

For the second part, by assumption, we have $${\mathbb{P}}_{\theta }(\theta \in C^{*}(𝐗))=1-\alpha .$$ So, the size of the test with the rejection region $R({\theta }_0)$ is $${\mathbb{P}}_{\theta ={\theta }_0}(𝐗\in R({\theta }_0))={\mathbb{P}}_{\theta ={\theta }_0}({\theta }_0=\theta \notin C^{*}(𝐗))=1-(1-\alpha )=\alpha .$$

$◻$