Statistics/Preliminaries

Template:Nav This chapter discusses some preliminary knowledge (related to statistics) for the following chapters in the advanced part.

Empirical distribution

Template:Colored definition Template:Colored remark Since all these $n$ random variables follow the same cdf as $X$, we may expect their distribution should be somewhat similar to the distribution of $X$, and indeed, this is true. Before showing how this is true, we need to define "the distribution of these $n$ random variables" more precisely, as follows: Template:Colored definition Template:Colored remark Template:Colored example Template:Colored remark Template:Colored theorem Template:Colored remark We have mentioned how we can approximate the cdf, and now we would like to estimate the Template:Colored em/Template:Colored em. Let us first discuss how to estimate the pmf.

For the discrete random variable $X$, from the empirical cdf, we know that each $X_1,\dots ,X_n$ is "assigned" with the probability $1/n$. Also, considering the previous example, the empirical pmf is $f_n(x)=\frac{\sum _{k=1}^n\mathrm{𝟏}\{X_k=x\}}{n}$. Template:Colored remark To discuss the estimation of pdf of continuous random variable, we need to define Template:Colored em first. Template:Colored definition For the continuous random variable $X$, construct class intervals for $X$ which are a non-overlapped partition of the interval $[X_{\text{min}},X_{\text{max}}]$, in which $X_{\text{min}}$ and $X_{\text{max}}$ are the minimum and maximum values in the sample. Then, the pdf $$f(x)\approx \frac{F(c_j)-F(c_{j-1})}{c_j-c_{j-1}},x\in (c_{j-1},c_j]\text{ and }j=1,2,\dots ,i,$$ when $c_{j-1}$ and $c_j$ are close, i.e. the length of each class interval is small. (Although the union of the above class intervals is $(c_0,c_i]$ and thus the value $c_0$ is not included in the interval, it does not matter since the value of the pdf at $c_0$ does not affect the calculation of probability.) Here, $c_0$ is $X_{\text{min}}$ and $c_i$ is $X_{\text{max}}$.

Since $F(c_j)-F(c_{j-1})=\mathbb{P}(X\in (c_{j-1},c_j])\approx \frac{\sum _{k=1}^n\mathrm{𝟏}\{X_k\in (c_{j-1},c_j]\}}{n}$ is the relative frequency of occurrences of the event $\{X_k\in (c_{j-1},c_j]\}$, we can rewrite the above expression as $$f(x)\approx h_n(x)=\frac{\sum _{k=1}^n\mathrm{𝟏}\{X_k\in (c_{j-1},c_j]\}}{n(c_j-c_{j-1})},x\in (c_{j-1},c_j]\text{ and }j=1,2,\dots ,i$$ in which $h_n(x)$ is called the Template:Colored em.

Since there are many possible ways to construct the class intervals, the value of $h_n(x)$ can differ even with the same $n$ and $x$. When $n$ is Template:Colored em and the length of each class interval is Template:Colored em, we will expect $h_n(x)$ to be a good estimate of $f(x)$ (the theoretical pdf).

There are some properties related to the relative frequency histogram, as follows: Template:Colored proposition

Proof.

(i) Since the indicator function is nonnegative (its value is either 0 or 1), $n$ is positive, and $c_j>c_{j-1}$ so $c_j-c_{j-1}$ is positive, we have $h_n(x)\ge 0$ by definition.

(ii) $$\begin{array}{cc}\underset{c_0}{\overset{c_i}{\int }}{h}_n(x)dx& =\underset{c_0}{\overset{c_1}{\int }}{h}_n(x)dx+\underset{c_1}{\overset{c_2}{\int }}{h}_n(x)dx+\cdots +\underset{c_{i-1}}{\overset{c_i}{\int }}{h}_n(x)dx\\ & =\frac{1}{n}(\underset{c_0}{\overset{c_1}{\int }}\frac{\sum _{k=1}^n\mathrm{𝟏}\{X_k\in (c_0,c_1]\}}{c_1-c_0}dx+\underset{c_1}{\overset{c_2}{\int }}\frac{\sum _{k=1}^n\mathrm{𝟏}\{X_k\in (c_1,c_2]\}}{c_2-c_1}dx+\cdots +\underset{c_{i-1}}{\overset{c_i}{\int }}\frac{\sum _{k=1}^n\mathrm{𝟏}\{X_k\in (c_{i-1},c_i]\}}{c_i-c_{i-1}}dx)\\ & =\frac{1}{n}(\frac{\sum _{k=1}^n\mathrm{𝟏}\{X_k\in (c_0,c_1]\}}{c_1-c_0}\cdot (c_1-c_0)+\frac{\sum _{k=1}^n\mathrm{𝟏}\{X_k\in (c_1,c_2]\}}{c_2-c_1}\cdot (c_2-c_1)+\cdots +\frac{\sum _{k=1}^n\mathrm{𝟏}\{X_k\in (c_{i-1},c_i]\}}{c_i-c_{i-1}}\cdot (c_i-c_{i-1}))\\ & =\frac{1}{n}(\sum _{k=1}^n\mathrm{𝟏}\{X_k\in (c_0,c_1]\}+\sum _{k=1}^n\mathrm{𝟏}\{X_k\in (c_1,c_2]\}+\cdots +\sum _{k=1}^n\mathrm{𝟏}\{X_k\in (c_{i-1},c_i]\})\\ & =\frac{1}{n}(\sum _{k=1}^n\mathrm{𝟏}\{X_k\in (c_0,c_1]\cup (c_1,c_2]\cup \cdots \cup (c_{i-1},c_i]\})\\ & =\frac{1}{n}(\sum _{k=1}^n\mathrm{𝟏}\{X_k\in \underset{\text{sample space of }X}{\underbrace{(c_0,c_i]}}\})\\ & =\frac{1}{n}\cdot \sum _{k=1}^{n}1\\ & =\frac{1}{n}\cdot n\\ & =1.\end{array}$$ Here, $c_0$ is $X_{\text{min}}$ and $c_i$ is $X_{\text{max}}$.

(iii) We can "split" the integral in a similar way as in (ii), and then eventually the integral equals $\frac{1}{n}\cdot \sum _{k=1}^n\mathrm{𝟏}\{X_k\in A\}$, and it can can approximate $\mathbb{P}(A)$ since it is the relative frequency of occurrences of the event $\{X_k\in A\}$.

$◻$

Expectation

In this section, we will discuss some results about expectation, which involve some sort of inequalities. Let $a$ and $b$ be constants. Also, let $\mathrm{\Omega }$ be the sample space of $X$.

Proof. Assume $\mathbb{P}(a<X\le B)=1$.

Case 1: $X$ is discrete.

By definition of expectation, $𝔼[X]=\sum _{x\in \mathrm{\Omega }}^{}xf(x)$. Then, we have $\sum _{x\in \mathrm{\Omega }}^{}af(x)<\sum _{x\in \mathrm{\Omega }}^{}xf(x)\le \sum _{x\in \mathrm{\Omega }}^{}bf(x)\Rightarrow a\sum _{x\in \mathrm{\Omega }}^{}f(x)<𝔼[X]\le b\sum _{x\in \mathrm{\Omega }}^{}f(x)\Rightarrow a<𝔼[X]\le b$ because of the condition $\mathbb{P}(a<X\le b)=1$.

Case 2: $X$ is continuous.

We have similarly $\underset{\mathrm{\Omega }}{\overset{}{\int }}af(x)dx<\underset{\mathrm{\Omega }}{\overset{}{\int }}xf(x)dx\le \underset{\mathrm{\Omega }}{\overset{}{\int }}bf(x)dx\Rightarrow a<𝔼[X]\le b$ because of the condition of $\mathbb{P}(a<X\le b)=1$.

$◻$

Proof. $$\frac{𝔼[X]}{a}=\frac{1}{a}\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\underset{\ge 0}{\underbrace{xf(x)}}dx\ge \underset{a}{\overset{\mathrm{\infty }}{\int }}xf(x)dx\ge \frac{1}{a}\underset{a}{\overset{\mathrm{\infty }}{\int }}af(x)dx=\underset{a}{\overset{\mathrm{\infty }}{\int }}f(x)dx=\mathbb{P}(X\ge a),$$ as desired.

$◻$

Proof. First, observe that $X^2$ is a nonnegative random variable. Then, by Markov's inequality, for each (positive) $a^{\prime }=a^2$, we have $$\mathbb{P}(X^2\ge a^{\prime })\le \frac{𝔼[X^2]}{a^{\prime }}⟹\mathbb{P}(X^2\ge a^2)\le \frac{𝔼[X^2]}{a^2}⟹\mathbb{P}(\sqrt{X^2}\ge \sqrt{a^2})\le \frac{𝔼[X^2]}{a^2}⟹\mathbb{P}(|X|\ge a)\le \frac{𝔼[X^2]}{a^2}$$, since $a$ is positive.

$◻$

Proof. Let $L(x)=a+bx$ be the tangent of the function $g(x)$ at $x=𝔼[X]$. Then, since $g$ is convex, we have $g(x)\ge L(x)$ for each $x$ (informally, we can observe this graphically). As a result, we have $$\begin{array}{cccc}& & \underset{\mathrm{\Omega }}{\overset{}{\int }}g(x)f(x)dx& \ge \underset{\mathrm{\Omega }}{\overset{}{\int }}L(x)f(x)dx\\ & \Rightarrow & 𝔼[g(X)]& \ge 𝔼[L(X)]\\ & & & =𝔼[a+bX]\\ & & & =a+b𝔼[X]\\ & & & =L(𝔼[X])\\ & & & =g(𝔼[X])& \text{since }L(x)\text{ is tangent of }g(x)\text{ at }x=𝔼[X],\end{array}$$ as desired.

$◻$

Proof. $$\begin{array}{cc}0& \le 𝔼[(X𝔼[Y^2]-Y𝔼[XY]{)}^2]\\ & =𝔼[X^2\underset{\text{constant}}{\underbrace{(𝔼[Y^2]{)}^2}}-2XY\underset{\text{constant}}{\underbrace{𝔼[Y^2]𝔼[XY]}}+Y^2\underset{\text{constant}}{\underbrace{(𝔼[XY]{)}^2}}]\\ & =(𝔼[Y^2]{)}^2𝔼[X^2]-2𝔼[Y^2]𝔼[XY]𝔼[XY]+(𝔼[XY]{)}^2𝔼[Y^2]\\ & =𝔼[Y^2](𝔼[X^2]𝔼[Y^2]-2(𝔼[XY]{)}^2+(𝔼[XY]{)}^2)\\ & =𝔼[Y^2](𝔼[X^2]𝔼[Y^2]-(𝔼[XY]{)}^2)\\ \end{array}$$ Since $𝔼[Y^2]\ge 0$, we must have $𝔼[X^2]𝔼[Y^2]-(𝔼[XY]{)}^2\ge 0\iff (𝔼[XY]{)}^2\le 𝔼[X^2]𝔼[Y^2]$.

$◻$

Convergence

Before discussing convergence, we will define some terms that will be used later. Template:Colored definition Template:Colored remark In a Template:Colored em, say $x_1,\dots ,x_n$, we observe Template:Colored em values of their sample mean, $\bar{x}=\frac{\sum _{i=1}^n{x}_i}{n}$, and sample variance, $s^2=\frac{\sum _{i=1}^n(x_i-\bar{x}{)}^2}{n}$. Template:Colored em, each of the values is only Template:Colored em realization of the respective random variables $\bar{X}$ and $S^2$. We should notice the difference between these definite values (not random variables) and the statistics (random variables).

To explain the definitions of the sample mean $\bar{X}$ and sample variance $S^2$ more intuitively, consider the following.

Recall that the empirical cdf $F_n(x)$ assigns probability $\frac{1}{n}$ to each of the random sample $X_1,\dots ,X_n$. Thus, by the definition of mean and variance, the Template:Colored em of a random variable, say $Y$, with this cdf $F_n(x)$ (and hence with the corresponding pmf $f_n(x)$) is $\sum _{i=1}^n(X_i\cdot \frac{1}{n})=\bar{X}$. Similarly, the Template:Colored em of $Y$ is $\sum _{i=1}^n((X_i-\bar{X}{)}^2\cdot \frac{1}{n})=S^2$. In other words, the Template:Colored em and Template:Colored em of the empirical distribution, which corresponds to the Template:Colored em, is the Template:Colored em $\bar{X}$ and the Template:Colored em $S^2$ respectively, which is quite natural, right? Template:Colored remark Also, recall that the empirical cdf $F_n(x)$ can well approximate the cdf of $X$, $F(x)$ when $n$ is large. Since $\bar{X}$ and $S^2$ are the mean and variance of a random variable with cdf $F_n(x)$ it is natural to expect that $\bar{X}$ and $S^2$ can well approximate the mean and variance of $X$.

Convergence in probability

Template:Colored definition Template:Colored remark The following theorem, namely Template:Colored em, is an important theorem which is related to convergence in probability. Template:Colored theorem

Proof. We use $S_n$ to denote $\sum _{i=1}^n{X}_i$.

By definition, $\bar{X}\stackrel{p}{\to }\mu$ as $n\to \mathrm{\infty }$ is equivalent to $\mathbb{P}(|\frac{S_n}{n}-\mu |>\epsilon )\to 0$ as $n\to \mathrm{\infty }$.

By Chebyshov's inequality, we have $$\begin{array}{cc}\mathbb{P}(|\frac{S_n}{n}-\mu |>ϵ)& \le \frac{1}{{\epsilon }^2}𝔼\left[{(\frac{S_n}{n}-\mu )}^2\right]\\ & =\frac{1}{{\epsilon }^2}𝔼\left[{\left(\frac{S_n-n\mu }{n}\right)}^2\right]\\ & =\frac{1}{n^2{\epsilon }^2}𝔼\left[{(S_n-n\mu )}^2\right]\\ & =\frac{1}{n^2{\epsilon }^2}𝔼\left[{(\sum _{i=1}^n{X}_i-\mu )}^2\right]\\ & =\frac{1}{n^2{\epsilon }^2}𝔼[\sum _{i=1}^n\sum _{j=1}^n(X_i-\mu )(X_j-\mu )]\\ & =\frac{1}{n^2{\epsilon }^2}(𝔼[\sum _{i=j=1}^n(X_i-\mu {)}^2]+𝔼[\sum _{i=1}^n\sum _{j\ne i,j=1}^n(X_i-\mu )(X_j-\mu )])\\ \end{array}$$

Since $X_1,X_2,\dots$ are Template:Colored em (and hence functions of them are also independent) and the expectation is multiplicative under independence, $$\begin{array}{cc}\frac{1}{n^2{\epsilon }^2}(𝔼[\sum _{i=j=1}^n(X_i-\mu {)}^2]+𝔼[\sum _{i=1}^n\sum _{j\ne i,j=1}^n(X_i-\mu )(X_j-\mu )])& =\frac{1}{n^2{\epsilon }^2}(𝔼[\sum _{i=j=1}^n(X_i-\mu {)}^2]+\sum _{i=1}^n\sum _{j\ne i,j=1}^n\underset{=\mu -\mu =0}{\underbrace{𝔼[X_i-\mu ]}}\underset{=\mu -\mu =0}{\underbrace{𝔼[X_j-\mu ]}})\\ & =\frac{1}{n^2{\epsilon }^2}\cdot \sum _{i=1}^n\underset{={\sigma }^2}{\underbrace{𝔼[(X_i-\mu {)}^2]}}\\ & =\frac{n{\sigma }^2}{n^2{\epsilon }^2}\\ & =\frac{{\sigma }^2}{n{\epsilon }^2}\\ & \to 0& \text{as }n\to \mathrm{\infty }.\end{array}$$ So, the probability $\mathbb{P}(|\frac{S_n}{n}-\mu |>\epsilon )$ is Template:Colored em an expression that tends to be 0 as $n\to \mathrm{\infty }$. Since the probability is nonnegative ($\ge 0$), it follows that the probability also tends to be 0 as $n\to \mathrm{\infty }$.

$◻$

Template:Colored remark There are also some properties of convergence in probability that help us to determine a complex expression converges to what thing. Template:Colored proposition

Proof. Template:Colored em: Assume $X_n\stackrel{p}{\to }X$ and $Y_n\stackrel{p}{\to }Y$. Continuous mapping theorem is first proven so that we can use it in the proof of other properties (the proof is omitted here). Also, it can be shown that $(X_n,Y_n)\stackrel{p}{\to }(X,Y)$ (joint convergence in probability, the definition is similar, except that the random variables become ordered pairs, so the interpretation of "$|Z_n-Z|$" becomes the Template:Colored em between the two points in Cartesian coordinate system, which are represented by the ordered pairs)

After that we define $g(z_1,z_2)=a{z}_1+b{z}_2$, $g(z_1,z_2)=z_1{z}_2$, and $g(z_1/z_2)=z_1/z_2$ respectively, where each of these functions is continuous, and $a,b$ are constants. Then, applying the continuous mapping theorem using each of these functions gives us the first three results.

$◻$

Convergence in distribution

Template:Colored definition Template:Colored remark A very important theorem in statistics which is related to convergence in distribution is Template:Colored em. Template:Colored theorem

Proof. A (lengthy) proof can be founded in Probability/Transformation of Random Variables#Central limit theorem.

$◻$

There are some properties of convergence in distribution, but they are a bit different from the properties of convergence in probability. These properties are given by Template:Colored em, and also continuous mapping theorem. Template:Colored theorem

Proof. Omitted.

$◻$

Proof. Template:Colored em: Assume $X_n\stackrel{d}{\to }X$ and $Y_n\stackrel{p}{\to }c$. Then, it can be shown that $(X_n,Y_n)\stackrel{d}{\to }(X,c)$ (joint convergence in distribution, and the definitions of this is similar, except that the cdf's become joint cdf's of ordered pairs). After that, we define $g(z_1,z_2)=z_1+z_2$,$g(z_1,z_2)=z_1{z}_2$, and $g(z_1,z_2)=z_1/z_2$ respectively, where each of the functions is continuous, and then applying the continuous mapping theorem using each of these functions gives us the three desired results.

$◻$

Resampling

By Template:Colored em, we mean creating new samples based on an existing sample. Now, let us consider the following for a general overview of the procedure of resampling.

Suppose $X_1,\dots ,X_n$ is a Template:Colored em from a distribution of a random variable $X$ with cdf, $F(x)$. Let $x_1,\dots ,x_n$ be a corresponding Template:Colored em of the random sample $X_1,\dots ,X_n$. Based on this realization, we have also a Template:Colored em of the empirical cdf: $\frac{1}{n}\sum _{k=1}^n\mathrm{𝟏}\{x_k\le x\}$ . Since this is a realization of the empirical cdf, by Glivenko-Cantelli theorem, it is a good estimate of the cdf $F(x)$ when $n$ is large . In other words, if we denote the random variable with the same pdf as that Template:Colored em of the empirical cdf by $X^{*}$, $X^{*}$ and $X$ have similar distributions when $n$ is large.

Notice that a realization of empirical cdf is a Template:Colored em cdf (since the support $x_1,\dots ,x_n$ is countable). We now draw a Template:Colored em (called the bootstrap (or resampling) random sample) with sample size $B$ (called the Template:Colored em) $X_1^{*},\dots ,X_B^{*}$ from the distribution of a random variable $X^{*}$ ($X^{*}$ comes from Template:Colored em from $X$, so the behaviour of sampling from $X^{*}$ is called Template:Colored em).

Then, the relative frequency historgram of $X_1^{*},\dots ,X_B^{*}$ should be close to that of the corresponding Template:Colored em of the empirical pmf of $X^{*}$ (found from the realization of the empirical cdf of $X^{*}$), which is close to pdf $f(x)$ of $X$. This means the relative frequency historgram of $X_1^{*},\dots ,X_B^{*}$ is close to the pdf $f(x)$ of $X$.

In particular, since the cdf of $X^{*}$, $F_n(x)$, assigns probability $1/n$ to each of $X_1^{*},\dots ,X_B^{*}$ , the pmf of $X^{*}$ is $\mathbb{P}(X^{*}=x_i)=\frac{1}{n},i=1,2,\dots ,n$. Notice that this pmf is quite simple, and therefore it can make the related calculation about it simpler. For example, in the following, we want to know the distribution of $T^{*}=g(X_1^{*},\dots ,X_n^{*})$, and this simple pmf can make the resulting distribution also quite simple.

Template:Colored remark In the following, we will discuss an application of the bootstrap method (or Template:Colored em) mentioned above, namely using bootstrap method to Template:Colored em the distribution of a statistic $T=g(X_1,X_2,\dots ,X_n)$ (the inputs of the functions are random variables and $g$ is a function). The reason for approximating, rather than finding the distribution exactly, is that the latter is usually infeasible (or may be too complicated).

To do this, consider the "bootstrapped statistic" $T^{*}=g(X_1^{*},X_2^{*},\dots ,X_n^{*})$ and the statistic $T=g(X_1,X_2,\dots ,X_n)$. $X_1^{*},X_2^{*},\dots ,X_n^{*}$ is the bootstrap random sample (with bootstrap sample size $n$) from the distribution of $X^{*}$ and $X_1,X_2,\dots ,X_n$ is the random sample from the distribution of $X^{*}$. When $n$ is large, since the distribution of $X^{*}$ is similar to that of $X$, the bootstrap random sample $X_1^{*},X_2^{*},\dots ,X_B^{*}$ and the random sample $X_1,X_2,\dots ,X_n$ are also similar. It follows that $T^{*}$ and $T$ are similar as well, or to be more precise, the Template:Colored em of $T^{*}$ and $T$ are close. As a result, we can utilize the distribution of $T^{*}$ (which is easier to find and simpler, since the pmf of $X^{*}$ is simple as in above) to approximate the distribution of $T$. A procedure to do this is as follows:

1. Generate a Template:Colored em $x_1^{*},x_2^{*},\dots ,x_n^{*}$ from the Template:Colored em $X_1^{*},X_2^{*},\dots ,X_n^{*}$, which is from the distribution of $X^{*}$.
2. Calculate a realization of the bootstrapped statistic $T^{*}$, $t^{*}=g(x_1^{*},x_2^{*},\dots ,x_n^{*})$.
3. Repeat 1. to 2. $j$ times to get a sequence of $j$ realizations of $T^{*}$: $t_1^{*},t_2^{*},\dots ,t_j^{*}$.
4. Plot the relative frequency historgram of the $j$ realizations $t_1^{*},t_2^{*},\dots ,t_j^{*}$.

This histogram of the $j$ realizations (which are a realization of a random sample from $T^{*}$ with sample size $j$) is close to the pmf of $T^{*}$ , and thus close to the pmf of $T$. Template:Nav Template:BookCat