Probability/Properties of Distributions

Introduction

Recall that pdf (or cdf) describes the random behaviour of a random variable Template:Colored em . However, we may sometimes find the pdf (or cdf) to be too complicated, and only want to know some Template:Colored em about the random variable. In view of this, we study some properties of distributions in this chapter, which provide Template:Colored em descriptions of the random behaviour of the random variable.

Some examples of such partial descriptions include

• location (e.g. pdf is 'located' at left or right?),
• dispersion (e.g. 'sharp' of 'flat' pdf?),
• skewness (e.g. pdf is symmetric, skewed to left, or skewed to right?), and
• tail property (e.g. pdf have 'light' or 'heavy' tails?).

We can Template:Colored em describe them, but such descriptions are quite subjective and inaccurate. To give a more objective and accurate measure to such descriptions, we evaluate them Template:Colored em using some quantitative measures derived from the pdf (or cdf) of the random variable.

We will discuss some of such quantitative measures in this chapter. Among these, the Template:Colored em is the most important one, since many of other properties base upon the concept of Template:Colored em.

Expectation

We have different alternative names for expectation, e.g. expected value and mean. Template:Colored definition Template:Colored remark Template:Colored example Template:Colored example Template:Colored exercise In the following, we introduce a useful result that gives the relationship between expectation and probability, we can use expectation to ease the computation of probability using this result. Template:Colored proposition

Proof. Let $X=\mathrm{𝟏}\{E\}$. Since $X=\mathrm{𝟏}\{E\}\sim \mathrm{Ber}(\mathbb{P}(E))$ (which is a discrete random variable), $$𝔼[X]=0[\mathbb{P}(X=0)]+1[\mathbb{P}(X=1)]=\mathbb{P}(\mathrm{𝟏}\{E\}=1)=\mathbb{P}(E).$$

$◻$

When there are multiple random variables involved, we may derive the joint pmf or pdf first to compute the expectation, but it can be quite difficult and complicated to do so. Practically, we use the following theorem more often. Template:Colored theorem Template:Colored remark The proof is quite complicated, and hence we skip it. In the following, we will introduce several properties of expectation that can help us to simplify computations of the expectation.

Proof.

Template:Colored em:

for continuous random variables $X,Y$, $$\begin{array}{cc}𝔼[\alpha X+\beta Y+\gamma ]=\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}(\alpha x+\beta y+\gamma )\underset{\text{joint pdf}}{\underbrace{f(x,y)}}dxdy& =\alpha \underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}x\underset{\text{marginal pdf of }X}{\underbrace{\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}f(x,y)dy}}dx+\beta \underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}y\underset{\text{margianl pdf of }Y}{\underbrace{\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}f(x,y)dx}}dy+\gamma \underset{1}{\underbrace{\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}f(x,y)dxdy}}\\ & =\alpha \underset{𝔼[X]}{\underbrace{\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}x{f}_X(x)dx}}+\beta \underset{𝔼[Y]}{\underbrace{\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}y{f}_Y(y)dy}}+\gamma \\ & =\alpha 𝔼[X]+\beta 𝔼[Y]+\gamma .\end{array}$$ Similarly, for discrete random variables $X,Y$, $$\begin{array}{cc}𝔼[\alpha X+\beta Y+\gamma ]& =\sum _x^{}\sum _y^{}(\alpha x+\beta y+\gamma )f(x,y)\\ & =\alpha \sum _x^{}x\sum _y^{}f(x,y)+\beta \sum _y^{}y\sum _x^{}f(x,y)+\gamma \sum _x^{}\sum _y^{}f(x,y)\\ & =\alpha \sum _x^{}{f}_X(x)+\beta \sum _y^{}{f}_Y(y)+\gamma (1)\\ & =\alpha 𝔼[X]+\beta 𝔼[Y]+\gamma .\end{array}$$

Template:Colored em:

For continuous random variable $X$, $$\underset{\because X\ge 0}{\underbrace{\underset{0}{\overset{\mathrm{\infty }}{\int }}}}x{f}_X(x)\ge 0.$$ Similarly, for discrete random variable $X$, $$\underset{\because X\ge 0}{\underbrace{\sum _{x\ge 0}^{}}}x{f}_X(x)\ge 0.$$

Template:Colored em:

For random variables $X,Y$ that are either both discrete or both continuous, $$X\ge Y\Rightarrow X-Y\ge 0\Rightarrow 𝔼[X]-𝔼[Y]\stackrel{\text{linearity}}{=}𝔼[X-Y]\stackrel{\text{nonnegativity}}{\ge }0.$$

Template:Colored em:

$$-|X|\le X\le |X|\stackrel{\text{monotonicity}}{\Rightarrow }-𝔼[|X|]\le 𝔼[X]\le 𝔼[|X|]$$

Template:Colored em:

For continuous random variables $X,Y$, $$𝔼[XY]=\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}xy\underset{\text{joint pdf}}{\underbrace{f(x,y)}}dxdy=\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}xy\underset{\text{marginal pdf's}}{\underbrace{f_X(x)f_Y(y)}}dxdy=\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}y{f}_Y(y)\underset{\text{independent from }y}{\underbrace{\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}x{f}_X(x)dx}}dy=\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}x{f}_X(x)dx\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}y{f}_Y(y)dy=𝔼[X]𝔼[Y].$$ Similarly, for discrete random variables $X,Y$, $$𝔼[XY]=\sum _x^{}\sum _y^{}xy\underset{\text{joint pmf}}{\underbrace{f(x,y)}}=\sum _y^{}\sum _x^{}xy\underset{\text{marginal pmf's}}{\underbrace{f_X(x)f_Y(y)}}=\left(\sum _x^{}x{f}_X(x)\right)\left(\sum _y^{}y{f}_Y(y)\right)=𝔼[X]𝔼[Y].$$

$◻$

Mean of some distributions of a discrete random variable

Proof.

• $𝔼[X]=\underset{=0}{\underbrace{0\cdot \mathbb{P}(X=0)}}+1\cdot \underset{=p}{\underbrace{\mathbb{P}(X=1)}}=p$.
• Since $Y=X_1+\cdots +X_n$, in which $X_1,\dots ,X_n$ are i.i.d. and follow $\mathrm{Ber}(p)$ ,
• $𝔼[Y]=𝔼[X_1+\cdots +X_n]\stackrel{\text{ linearity }}{=}𝔼[X_1]+\cdots +𝔼[X_n]=\underset{n\text{ times}}{\underbrace{p+\cdots +p}}=np$.

$◻$

Proof. $$𝔼[X]=\sum _{k=0}^{\mathrm{\infty }}k\underset{\mathbb{P}(X=k)}{\underbrace{\left(\frac{{\lambda }^k{e}^{-\lambda }}{k!}\right)}}=\lambda (0+\sum _{\underset{k-1=0}{\underbrace{k=1}}}^{\mathrm{\infty }}\underset{\mathbb{P}(X=k-1)}{\underbrace{\cancel{k}\left(\frac{{\lambda }^{k-1}{e}^{-\lambda }}{\cancel{k}(k-1)!}\right)}})=\lambda (0+1)=\lambda .$$

$◻$

Proof.

• Since

$$\begin{array}{cc}𝔼[X]& =\sum _{k=0}^{\mathrm{\infty }}k\underset{\mathbb{P}(X=k)}{\underbrace{(1-p{)}^{k}p}}\\ & =\sum _{k=0}^{\mathrm{\infty }}(k-1)(1-p{)}^{k}p+\stackrel{=1}{\overbrace{\sum _{k=0}^{\mathrm{\infty }}\underset{\mathbb{P}(X=k)}{\underbrace{(1-p{)}^{k}p}}}}\\ & =\underset{=-p}{\underbrace{(0-1)(1-p{)}^{0}p}}+((1-p)\sum _{k-1=0}^{\mathrm{\infty }}(k-1)(1-p{)}^{k-1}p)+1\\ & =-p+(1-p)𝔼[X]+1,\\ \end{array}$$

• it follows that $p𝔼[X]=1-p\Rightarrow 𝔼[X]=\frac{1-p}{p}.$.
• Since $Y=X_1+\cdots +X_k$ in which $X_1,\dots ,X_k$ are i.i.d., and follow $\mathrm{Geo}(p)$ ,
• $𝔼[Y]=𝔼[X_1]+\cdots +𝔼[X_k]=\underset{k\text{ times}}{\underbrace{\frac{1-p}{p}+\cdots +\frac{1-p}{p}}}=\frac{k(1-p)}{p}.$

$◻$

Proof.

• Since $X=X_1+\cdots +X_n$ in which $X_1,\dots ,X_n\sim \mathrm{Ber}(K/N)$ (each of the Bernoulli r.v.'s indicates whether the corresponding draw of ball is of type 1, with probability $K/N$ without knowing the results of other draws , since each draw is equally likely to be any of the $N$ balls) ,
• it follows that $𝔼[X]=𝔼[X_1]+\cdots +𝔼[X_n]=\underset{n\text{ times}}{\underbrace{\frac{K}{N}+\cdots +\frac{K}{N}}}=\frac{nK}{N}.$

$◻$

Mean of some distributions of a continuous random variable

We will introduce the formulas for mean of some distributions of a Template:Colored em random variable, which are relatively simpler. Template:Colored proposition

Proof. $$𝔼[X]=\underset{a}{\overset{b}{\int }}\frac{x}{b-a}dx=\frac{1}{2(b-a)}(b^2-a^2)=\frac{\cancel{(b-a)}(b+a)}{2\cancel{(b-a)}}.$$

$◻$

Proof.

• It suffices to prove the formula for mean of gamma r.v.'s, since exponential and chi-squared r.v.'s are essentially special cases of gamma r.v.'s, and thus we can simply substitute some values into the formula for mean of gamma r.v.'s to obtain the formulas for them.
• $$\begin{array}{cc}𝔼[X]& =\underset{0}{\overset{\mathrm{\infty }}{\int }}x\cdot \frac{{\lambda }^{\alpha }{x}^{\alpha -1}{e}^{-\lambda x}}{\mathrm{\Gamma }(\alpha )}dx\\ & =\frac{\alpha }{\lambda }\underset{=F(\mathrm{\infty })=1}{\underbrace{\underset{0}{\overset{\mathrm{\infty }}{\int }}\frac{{\lambda }^{\alpha +1}{x}^{\alpha +1-1}{e}^{-\lambda x}}{\mathrm{\Gamma }(\alpha +1)}dx}},& F\text{ is the cdf of }\mathrm{Gamma}(\alpha +1,\lambda ),\\ & =\frac{\alpha }{\lambda }.\\ \end{array}$$
• Since $\mathrm{Exp}(\lambda )\equiv \mathrm{Gamma}(1,\lambda )$, $𝔼[Y]=1/\lambda$ by substituting $\alpha =1$.
• Since ${\chi }_{\nu }^2\equiv \mathrm{Gamma}(\nu /2,1/2)$, $𝔼[Z]=(\nu \cancel{/2})/\cancel{(1/2)}=\nu$ by substituting $\alpha =\nu /2$ and $\lambda =1/2$.

$◻$

Proof.

• We use similar approach from the previous proof.

$$\begin{array}{cc}𝔼[X]& =\underset{0}{\overset{1}{\int }}x\cdot \frac{\mathrm{\Gamma }(\alpha +\beta )}{\mathrm{\Gamma }(\alpha )\mathrm{\Gamma }(\beta )}{x}^{\alpha -1}(1-x{)}^{\beta -1}dx\\ & =\frac{\alpha }{\alpha +\beta }\underset{F(1)=1}{\underbrace{\underset{0}{\overset{1}{\int }}\frac{\mathrm{\Gamma }(\alpha +\beta +1)}{\mathrm{\Gamma }(\alpha +1)\mathrm{\Gamma }(\beta )}{x}^{\alpha +1-1}(1-x{)}^{\beta -1}dx}},& F\text{ is the cdf of }\mathrm{Beta}(\alpha +1,\beta ),\\ & =\frac{\alpha }{\alpha +\beta }.\end{array}$$

$◻$

Proof. $$\begin{array}{ccc}𝔼[X]& =𝔼[X-\theta ]+\theta & \text{by linearity},\\ & =\theta +\frac{1}{\pi }\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}(x-\theta )\cdot \frac{1}{1+(x-\theta {)}^2}dx\\ & =\theta +\frac{1}{\pi }\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\frac{u}{1+u^2}du,& \text{let }u=x-\theta \Rightarrow du=dx,\\ & =\theta +\frac{1}{\pi }(\underset{-\mathrm{\infty }}{\overset{0}{\int }}\frac{u}{1+u^2}du+\underset{0}{\overset{\mathrm{\infty }}{\int }}\frac{u}{1+u^2}du)\\ & =\theta +\frac{1}{\pi }(\frac{1}{2}[\ln (1+u^2){]}_{u=-\mathrm{\infty }}^{u=0}+\frac{1}{2}[\ln (1+u^2){]}_{u=0}^{u=\mathrm{\infty }})\\ & =\theta +\frac{1}{\pi }(\underset{\text{undefined}}{\underbrace{-\mathrm{\infty }+\mathrm{\infty }}}).\end{array}$$

$◻$

Proof.

• Let $Z=\frac{X-\mu }{\sigma }\sim 𝒩(0,1)$.
• $$\begin{array}{cc}𝔼[Z]& =\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}x\phi (x)dx\\ & =\frac{1}{\sqrt{2\pi }}\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}x{e}^{-x^2/2}dx\\ & =-\frac{1}{\sqrt{2\pi }}\underset{-\mathrm{\infty }}{\overset{-\mathrm{\infty }}{\int }}{e}^{u}du,& \text{let }u=-\frac{x^2}{2}\Rightarrow du=-dx\\ & =-\frac{1}{\sqrt{2\pi }}(\underset{=0}{\underbrace{e^{-\mathrm{\infty }}}}-\underset{=0}{\underbrace{e^{-\mathrm{\infty }}}})\\ & =0.\end{array}$$
• It follows that $𝔼[X]=𝔼[\sigma Z+\mu ]=\sigma \underset{=0}{\underbrace{𝔼[Z]}}+\mu =\mu$.

$◻$

Examples

Template:Colored example Template:Colored exercise Let us illustrate the usefulness of fundamental bridge between probability and expectation by giving a proof to inclusion-exclusion using this bridge. Template:Colored example

Probability generating functions

An application of expectation is Template:Colored em. As suggested by its name, it can Template:Colored em probabilities in some sense. Template:Colored definition Template:Colored remark

Variance (and standard deviation)

Indeed, Template:Colored em is a special case of Template:Colored em, and is related to Template:Colored em in some sense. Template:Colored definition Template:Colored definition Template:Colored definition Since $(X-𝔼[X]{)}^2$ is the squared deviation of the value of $X$ from its mean, in view of the definition of variance, we can see that variance measure the Template:Colored em (or Template:Colored em) of distribution, since it is what we would Template:Colored em of the squared deviation if we are to take an observation of the random variable.

Another term which is closed related is Template:Colored em. Template:Colored definition Template:Colored remark Template:Colored proposition

Proof.

• alternative expression for variance:

Let $\mu =𝔼[X]$ for clearer expression.

$$𝔼[(X-\mu {)}^2]=𝔼[X^2-2X\mu +{\mu }^2]=𝔼\left[X^2\right]-2\mu \underset{\mu }{\underbrace{𝔼[X]}}+{\mu }^2=𝔼\left[X^2\right]-{\mu }^2,$$ and the result follows.

• invariance under change in location parameter:

$$\mathrm{Var}(X+a)=𝔼[(X\cancel{+a}-\underset{𝔼[X]\cancel{+a}}{\underbrace{𝔼[X+a]}}{)}^2]=𝔼[(X-𝔼[X]{)}^2]=\mathrm{Var}(X).$$

• nonnegativity: it follows from $(X-𝔼[X]{)}^2\ge 0$.
• zero variance implies non-randomness:

Let $\mu =𝔼[X]$ for clearer expression. Consider the event $E_n=\{|X-\mu |\ge n^{-1}\}$, in which $n$ is a positive integer.

Since $$0=\mathrm{Var}(X)=𝔼[(X-\mu {)}^2]\ge 𝔼[(X-\mu {)}^2\underset{\le 1}{\underbrace{\mathrm{𝟏}\{E_n\}}}]=𝔼[|X-a{|}^2\mathrm{𝟏}\{E_n\}]\ge 𝔼[\underset{\text{constant}}{\underbrace{n^{-2}}}\mathrm{𝟏}\{E_n\}]=\underset{\ge 0}{\underbrace{n^{-2}}}\underset{\ge 0}{\underbrace{\mathbb{P}(E_n)}}\ge 0,$$

we have $0\ge n^{-2}\mathbb{P}(E_n)\ge 0\Rightarrow 0\ge \mathbb{P}(E_n)\ge 0\Rightarrow \mathbb{P}(E_n)=0$.

Thus,

$$\mathbb{P}(\underset{X\ne \mu }{\underbrace{|X-\mu |>0}})=\mathbb{P}\left(\bigcup _{n=1}^{\mathrm{\infty }}{E}_n\right)\stackrel{\text{a lemma}}{=}\underset{n\to \mathrm{\infty }}{\lim }\underset{0}{\underbrace{\mathbb{P}(E_n)}}=0\Rightarrow \mathbb{P}(X=\mu )=1-\underset{0}{\underbrace{\mathbb{P}(X\ne \mu )}}=1$$

• additivity under independence:

For each random variable $X$ and $Y$ that are independent with means $\mu ,\nu$ respectively,

$$\begin{array}{cc}\mathrm{Var}(X+Y)& =𝔼[(X+Y-𝔼[X+Y]{)}^2]\\ \mathrm{Var}(X+Y)& =𝔼[(X+Y-\mu -\nu {)}^2]& \text{by linearity}\\ & =\underset{\mathrm{Var}(X)}{\underbrace{𝔼[(X-\mu {)}^2]}}+\underset{\mathrm{Var}(Y)}{\underbrace{𝔼[(Y-\nu {)}^2]}}+2𝔼[(X-\mu )(Y-\nu )]& \text{by linearity}\\ & =\mathrm{Var}(X)+\mathrm{Var}(Y)+2𝔼[XY]-2\nu 𝔼[X]-2\mu 𝔼[Y]+2\mu \nu & \text{by linearity}\\ & =\mathrm{Var}(X)+\mathrm{Var}(Y)+2\underset{\mu \nu }{\underbrace{𝔼[X]𝔼[Y]}}-2\nu \mu \cancel{-2\mu \nu }\cancel{+2\mu \nu }& \text{by independence of }X,Y\\ & =\mathrm{Var}(X)+\mathrm{Var}(Y)\cancel{+2\mu \nu }\cancel{-2\nu \mu }\\ & =\mathrm{Var}(X)+\mathrm{Var}(Y).\end{array}$$ Thus, inductively, $$\mathrm{Var}(X_1+\cdots +X_n)=\mathrm{Var}(X_1)+\mathrm{Var}(X_2+\cdots +X_n)=\cdots =\mathrm{Var}(X_1)+\cdots +\mathrm{Var}(X_n)$$ if $X_1,\dots ,X_n$ are independent.

$◻$

Variance of some distributions of a discrete random variable

Proof.

• $𝔼[X^2]=0\cdot \mathbb{P}(X=0)+1\cdot \mathbb{P}(\underset{\iff X=1}{\underbrace{X^2=1}})=p$ since $X$ is nonnegative.
• It follows that $\mathrm{Var}(X)=𝔼[X^2]-(𝔼[X]{)}^2=p-p^2=p(1-p)$.
• Similar to the proof for the mean of Bernoulli and binomial r.v.'s, $Y=X_1+\cdots +X_n$ in which $X_1,\dots ,X_n$ are i.i.d. and follow $\mathrm{Ber}(p)$.
• Because of the Template:Colored em (from i.i.d. property), $\mathrm{Var}(Y)=\underset{n\text{ times}}{\underbrace{\mathrm{Var}(X_1)+\cdots +\mathrm{Var}(X_n)}}=np(1-p).$

$◻$

Proof.

• $$𝔼[X^2]=\sum _{k=0}^{\mathrm{\infty }}{k}^2\underset{\mathbb{P}(X=k)}{\underbrace{\left(\frac{{\lambda }^k{e}^{-\lambda }}{k!}\right)}}=\lambda (0+\sum _{\underset{k-1=0}{\underbrace{k=1}}}^{\mathrm{\infty }}\cancel{k}\left(\frac{k{\lambda }^{k-1}{e}^{-\lambda }}{\cancel{k}(k-1)!}\right))=\lambda (\underset{𝔼[X]}{\underbrace{\sum _{k-1=0}^{\mathrm{\infty }}\frac{(k-1)e^{-\lambda }{\lambda }^{k-1}}{(k-1)!}}}+\stackrel{=1}{\overbrace{\sum _{k-1=0}^{\mathrm{\infty }}\underset{\mathbb{P}(X=k-1)}{\underbrace{\frac{e^{-\lambda }{\lambda }^{k-1}}{(k-1)!}}}}})=\lambda (\lambda +1).$$
• Hence, $\mathrm{Var}(X)=𝔼[X^2]-(𝔼[X]{)}^2=\lambda (\lambda +1)-{\lambda }^2=\lambda .$

$◻$

Proof.

• Since

$$\begin{array}{cc}𝔼[X]& =\sum _{k=0}^{\mathrm{\infty }}{k}^2\underset{\mathbb{P}(X=k)}{\underbrace{(1-p{)}^{k}p}}\\ & =\sum _{k=0}^{\mathrm{\infty }}(k-1+1{)}^2\underset{\mathbb{P}(X=k)}{\underbrace{(1-p{)}^{k}p}}\\ & =\sum _{k=0}^{\mathrm{\infty }}(k-1{)}^2(1-p{)}^{k}p+\sum _{k=0}^{\mathrm{\infty }}2(k-1)(1-p{)}^{k}p+\stackrel{=1}{\overbrace{\sum _{k=0}^{\mathrm{\infty }}\underset{\mathbb{P}(X=k)}{\underbrace{(1-p{)}^{k}p}}}}\\ & =\underset{=p}{\underbrace{(0-1{)}^2(1-p{)}^{0}p}}+(1-p)\sum _{k-1=0}^{\mathrm{\infty }}(k-1{)}^2(1-p{)}^{k-1}p+\underset{=-2p}{\underbrace{2(0-1)(1-p{)}^{0}p}}+2(1-p)\sum _{k-1=0}^{\mathrm{\infty }}(k-1)(1-p{)}^{k-1}p+1\\ & =p+(1-p)𝔼[X^2]-2p+2(1-p)\underset{(1-p)/p}{\underbrace{𝔼[X]}}+1\\ & =(1-p)𝔼[X^2]+\frac{2(1-p{)}^2}{p}+1-p,\\ \end{array}$$

• it follows that $p𝔼[X^2]=\frac{2(1-p{)}^2}{p}+1-p\Rightarrow 𝔼[X^2]=\frac{2(1-p{)}^2+p(1-p)}{p^2}$.
• Hence, $\mathrm{Var}(X)=𝔼[X^2]-(𝔼[X]{)}^2=\frac{2(1-p{)}^2+p(1-p)}{p^2}-\frac{(1-p{)}^2}{p^2}=\frac{(1-p{)}^2+p(1-p)}{p^2}=\frac{(1-p)(1\cancel{-p+p})}{p^2}$.
• Similarly, $Y=X_1+\cdots +X_k$ in which $X_1,\dots ,X_k$ are i.i.d., and follow $\mathrm{Geo}(p)$ .
• Because of the independence, $\mathrm{Var}(Y)=\mathrm{Var}(X_1)+\cdots +\mathrm{Var}(X_k)=\underset{k\text{ times}}{\underbrace{\frac{1-p}{p^2}+\cdots +\frac{1-p}{p^2}}}=\frac{k(1-p)}{p^2}.$

$◻$

Variance of some distributions of a continuous random variable

Proof. $$\begin{array}{cc}\mathrm{Var}(X)& =𝔼\left[X^2\right]-(𝔼[X]{)}^2\\ & =\underset{a}{\overset{b}{\int }}\frac{x^2}{b-a}dx-{\left(\frac{b+a}{2}\right)}^2\\ & =\frac{1}{b-a}(b^3/3-a^3/3)-{\left(\frac{a+b}{2}\right)}^2\\ & =\frac{1}{3(b-a)}(b^3-a^3)-{\left(\frac{a+b}{2}\right)}^2\\ & =\frac{1}{3\cancel{(b-a)}}\cancel{(b-a)}(b^2+ba+a^2)-\frac{a^2+2ab+b^2}{4}\\ & =\frac{\cancel{4}{b}^2\cancel{+4ab}+\cancel{4}{a}^2\cancel{-3b^2}-\stackrel{2}{\cancel{6}}ab\cancel{-3a^2}}{12}\\ & =\frac{b^2-2ab+a^2}{12}\\ & =\frac{(b-a{)}^2}{12}.\\ \end{array}$$

$◻$

Proof.

• Similarly, it suffices to prove the formula for variance of gamma r.v.'s.
• $$\begin{array}{cc}𝔼[X^2]& =\underset{0}{\overset{\mathrm{\infty }}{\int }}{x}^2\cdot \frac{{\lambda }^{\alpha }{x}^{\alpha -1}{e}^{-\lambda x}}{\mathrm{\Gamma }(\alpha )}dx\\ & =\frac{(\alpha +1)\alpha }{{\lambda }^2}\underset{=F(\mathrm{\infty })=1}{\underbrace{\underset{0}{\overset{\mathrm{\infty }}{\int }}\frac{{\lambda }^{\alpha +2}{x}^{\alpha +2-1}{e}^{-\lambda x}}{\mathrm{\Gamma }(\alpha +2)}dx}},& F\text{ is the cdf of }\mathrm{Gamma}(\alpha +2,\lambda ),\\ & =\frac{(\alpha +1)\alpha }{{\lambda }^2}.\\ \end{array}$$
• It follows that $\mathrm{Var}(X)=𝔼[X^2]-(𝔼[X{]}^2)=\frac{(\cancel{\alpha }+1)\alpha }{{\lambda }^2}\cancel{-\frac{{\alpha }^2}{{\lambda }^2}}=\frac{\alpha }{{\lambda }^2}.$
• Since $\mathrm{Exp}(\lambda )\equiv \mathrm{Gamma}(1,\lambda )$, $\mathrm{Var}(Y)=1/{\lambda }^2$ by substituting $\alpha =1$.
• Since ${\chi }_{\nu }^2\equiv \mathrm{Gamma}(\nu /2,1/2)$, $\mathrm{Var}(Z)=(\nu /2)/(1/2{)}^2=2\nu$ by substituting $\alpha =\nu /2$ and $\lambda =1/2$.

$◻$

Proof.

• $$\begin{array}{cc}𝔼[X^2]& =\underset{0}{\overset{1}{\int }}{x}^2\cdot \frac{\mathrm{\Gamma }(\alpha +\beta )}{\mathrm{\Gamma }(\alpha )\mathrm{\Gamma }(\beta )}{x}^{\alpha -1}(1-x{)}^{\beta -1}dx\\ & =\frac{(\alpha +1)\alpha }{(\alpha +\beta +1)(\alpha +\beta )}\underset{F(1)=1}{\underbrace{\underset{0}{\overset{1}{\int }}\frac{\mathrm{\Gamma }(\alpha +\beta +2)}{\mathrm{\Gamma }(\alpha +2)\mathrm{\Gamma }(\beta )}{x}^{\alpha +2-1}(1-x{)}^{\beta -1}dx}},& F\text{ is the cdf of }\mathrm{Beta}(\alpha +2,\beta ),\\ & =\frac{(\alpha +1)\alpha }{(\alpha +\beta +1)(\alpha +\beta )}.\end{array}$$
• It follows that $$\begin{array}{cc}\mathrm{Var}(X)& =𝔼[X^2]-(𝔼[X]{)}^2=\frac{(\alpha +1)\alpha }{(\alpha +\beta +1)(\alpha +\beta )}-\frac{{\alpha }^2}{(\alpha +\beta {)}^2}\\ & =\frac{(\alpha +1)(\alpha )(\alpha +\beta )-{\alpha }^2(\alpha +\beta +1)}{(\alpha +\beta {)}^2(\alpha +\beta +1)}\\ & =\frac{\alpha (\cancel{{\alpha }^2+\alpha \beta +\alpha }+\beta \cancel{-{\alpha }^2-\alpha \beta -\alpha })}{(\alpha +\beta {)}^2(\alpha +\beta +1)}\\ & =\frac{\alpha \beta }{(\alpha +\beta {)}^2(\alpha +\beta +1)}.\\ \end{array}$$

$◻$

Proof. It follows from the proposition about undefined mean of Cauchy r.v.'s and the formula $\mathrm{Var}(X)=𝔼[X^2]-(𝔼[X]{)}^2$ (arbitrary term minus undefined term is undefined).

$◻$

Proof.

• Let $Z=\frac{X-\mu }{\sigma }\sim 𝒩(0,1)$.
• $$\begin{array}{cc}𝔼[Z^2]& =\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}{x}^2\phi (x)dx\\ & =\frac{1}{\sqrt{2\pi }}\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}{x}^2{e}^{-x^2/2}dx\\ & =-\frac{1}{\sqrt{2\pi }}\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}xd(e^{-x^2/2})\\ & =-\frac{1}{\sqrt{2\pi }}([x{e}^{-x^2/2}{]}_{-\mathrm{\infty }}^{\mathrm{\infty }}-\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}{e}^{-x^2/2}dx)& \text{by integration by parts},\\ & =-\frac{1}{\sqrt{2\pi }}(0-0-\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}{e}^{-x^2/2}dx)& \text{since exponential function }\downarrow \text{ much faster than linear function, or by L'Hospital rule},\\ & =\underset{=\mathrm{\Phi }(\mathrm{\infty })=1}{\underbrace{\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\phi (x)dx}}\\ & =1.\end{array}$$
• It follows that $\mathrm{Var}(Z)=𝔼[Z^2]-(𝔼[Z]{)}^2=1-0=1$.
• Hence, $\mathrm{Var}(X)=\mathrm{Var}(\sigma Z+\mu )={\sigma }^2\mathrm{Var}(Z)={\sigma }^2$.

$◻$

Quantile

Then, we will discuss Template:Colored em. In particular, Template:Colored em and Template:Colored em range are quite related to Template:Colored em. Template:Colored definition Template:Colored remark The following are some terminologies related to Template:Colored em. Template:Colored definition Template:Colored example Template:Colored definition Template:Colored definition Template:Colored example Template:Colored definition Template:Colored em and Template:Colored em measure centrality and dispersion respectively. Recall that Template:Colored em and Template:Colored em measure the same things respectively. One advantage of Template:Colored em and Template:Colored em is Template:Colored em, since they are always defined, while Template:Colored em and Template:Colored em can be infinite, and they fail to measure centrality and dispersion in those occasions. However, Template:Colored em and Template:Colored em also have some disadvantages, e.g. they may be more difficult to be computed, and may not be very accurate. Template:Colored example Template:Colored exercise

Mode

Mode is another measure of centrality. Template:Colored definition Template:Colored remark Template:Colored example Template:Colored remark

Covariance and correlation coefficients

In this section, we will discuss two important properties of Template:Colored em distributions, namely Template:Colored em and Template:Colored em. As we will see, covariance is related to variance in some sense, and correlation coefficient is closed related to correlation. Template:Colored definition Template:Colored definition Both Template:Colored em and Template:Colored em measure Template:Colored em between $X$ and $Y$. As we will see, $\rho (X,Y)\in [-1,1]$, $X,Y$ are more highly correlated as $|\rho (X,Y)|$ increases, and $X$ has a linear relationship with $Y$ if $|\rho (X,Y)|=1$.

Proof.

(i) $$\mathrm{Cov}(X,Y)=𝔼[(X-𝔼[X])(Y-𝔼[Y])]=𝔼[(Y-𝔼[Y])(X-𝔼[X])]=\mathrm{Cov}(Y,X)$$ (ii) $$\mathrm{Cov}(X,X)=𝔼[(X-𝔼[X])(X-𝔼[X])]=𝔼[(X-𝔼[X]{)}^2]=\mathrm{Var}(X)$$ (iii) $$\begin{array}{cc}\mathrm{Cov}(X,Y)& =𝔼[(X-𝔼[X])(Y-𝔼[Y])]\\ & =𝔼[XY-X𝔼[Y]-Y𝔼[X]+𝔼[X]𝔼[Y]]\\ & =𝔼[XY]-𝔼[Y]𝔼[X]\cancel{-𝔼[X]𝔼[Y]+𝔼[X]𝔼[Y]}\text{by linearity}\\ & =𝔼[XY]-𝔼[X]𝔼[Y]\end{array}$$ (iv) $$\begin{array}{cc}\mathrm{Cov}(\sum _{i=1}^n(a_i{X}_i+c),\sum _{j=1}^m(b_j{Y}_j+d))& =𝔼\left[(\sum _{i=1}^n(a_i{X}_i+c)-\sum _{i=1}^n𝔼[a_i{X}_i+c])(\sum _{j=1}^m(b_j{Y}_j+d)-\sum _{j=1}^m𝔼[b_j{Y}_j+d])\right]\\ & =𝔼[\sum _{i=1}^n(a_i{X}_i-𝔼[a_i{X}_i])\sum _{j=1}^m(b_j{Y}_j-𝔼[b_j{Y}_j])]\\ & =𝔼[\sum _{i=1}^n\sum _{j=1}^m(a_i{X}_i-𝔼[a_i{X}_i])(b_j{Y}_j-𝔼[b_j{Y}_j])]\\ & =\sum _{i=1}^n\sum _{j=1}^m𝔼[(a_i{X}_i-a_i𝔼[X_i])(b_j{Y}_j-b_j𝔼[Y_j])]& \text{by linearity}\\ & =\sum _{i=1}^n\sum _{j=1}^m{a}_i{b}_j𝔼[X_i-𝔼[X_i]]𝔼[Y_j-𝔼[Y_j]]\\ & =\sum _{i=1}^n\sum _{j=1}^m{a}_i{b}_j\mathrm{Cov}(X_i,Y_j)\end{array}$$ (v) $$\begin{array}{cc}\mathrm{Var}\left(\sum _{i=1}^n{X}_i\right)& \stackrel{\text{(ii)}}{=}\mathrm{Cov}(\sum _{i=1}^n{X}_i,\sum _{j=1}^n{X}_j)\\ & \stackrel{\text{(iv)}}{=}\sum _{i=1}^n\sum _{j=1}^n\mathrm{Cov}(X_1,X_j)\\ & =\sum _{1\le i=j\le n}^{}\mathrm{Cov}(X_i,X_j)+\sum _{1\le i\ne j\le n}^{}\mathrm{Cov}(X_i,X_j)\\ & \stackrel{\text{(ii)}}{=}\sum _{i=1}^n\mathrm{Var}(X_i)+\sum _{1\le i<j\le n}^{}\mathrm{Cov}(X_i,X_j)+\sum _{1\le j<i\le n}^{}\mathrm{Cov}(X_i,X_j)\\ & \stackrel{\text{(i)}}{=}\sum _{i=1}^n\mathrm{Var}(X_i)+2\sum _{1\le i<j\le n}^{}\mathrm{Cov}(X_i,X_j)\end{array}$$

$◻$

Then, we will discuss about Template:Colored em. The following is the definition of Template:Colored em between correlation between two random variables. Template:Colored definition Template:Colored remark Covariance and correlation coefficient are Template:Colored em, but they have differences. In particular, $\mathrm{Cov}(X,Y)$ depends on Template:Colored em of $X$ and $Y$, not just their relationship. Thus, this number is affected by the variances, and does not measure their relationship accurately. On the other hand, $\rho (X,Y)$ Template:Colored em for Template:Colored em of $X$ and $Y$, and therefore measures their relationships more Template:Colored em.

The following is one of the most important properties of correlation coefficient. Template:Colored proposition

Proof. For each random variable $X,Y$,

Template:Colored em: prove that $\rho (X,Y)\le 1\iff \frac{\mathrm{Cov}(X,Y)}{\sqrt{\mathrm{Var}(X)\mathrm{Var}(Y)}}\le 1$. To get rid of the square root to make the proof neater, we square both side of the inequality, and get $\frac{\mathrm{Cov}(X,Y{)}^2}{\mathrm{Var}(X)\mathrm{Var}(Y)}\le 1\iff \frac{\mathrm{Cov}(X,Y{)}^2}{\mathrm{Var}(Y)}\le \mathrm{Var}(X)\iff \mathrm{Var}(X)+\frac{\mathrm{Cov}(X,Y{)}^2}{\mathrm{Var}(Y)}\ge 0$.

Recall that $\mathrm{Var}(\cdot )\ge 0$. So, one way to prove the rightmost inequality is expressing its left side as $\mathrm{Var}(\cdot )$, as follows: $$\mathrm{Var}(X)-\frac{\mathrm{Cov}(X,Y{)}^2)}{\mathrm{Var}(Y)}=\mathrm{Var}(X)+{\left(\frac{\mathrm{Cov}(X,Y))}{\mathrm{Var}(Y)}\right)}^2\mathrm{Var}(Y)-2\left(\frac{\mathrm{Cov}(X,Y)}{\mathrm{Var}(Y)}\right)\stackrel{\text{(iv,v)}}{=}\mathrm{Var}(X-\frac{\mathrm{Cov}(X,Y)}{\mathrm{Var}(Y)}Y).$$ Thus, the result follows.

$◻$

Template:Colored remark Then, we will define several terminologies related to correlation coefficient. Template:Colored definition

Then, we will state an important result that is related to independence and correlation. Intuitively, you may think that 'independent' is the same as 'uncorrelated'. However, this is wrong. Indeed, 'independent' is Template:Colored em than 'uncorrelated'. Template:Colored proposition

Proof. For each independent random variable $X,Y$ with mean $\mu ,\nu$ respectively, $$\mathrm{Cov}(X,Y)=𝔼[(X-\mu )(Y-\nu )]\stackrel{\text{independence}}{=}𝔼[X-\mu ]𝔼[Y-\nu ]\stackrel{\text{linearity}}{=}(\underset{\mu }{\underbrace{𝔼[X]}}-\mu )(\underset{\nu }{\underbrace{𝔼[Y]}}-\nu )=0$$

$◻$

However, converse is Template:Colored em true, as we will see in the following example. Template:Colored example Template:Colored exercise Template:Nav