Probability/Important Distributions

Distributions of a discrete random variable

Binomial distribution

Motivation

Consider $n$ independent Bernoulli trials with the same success probability $p$. We would like to calculate to probability $\mathbb{P}(\{r\text{ successes in }n\text{ trials}\})$.

Let $S_i$ be the event $\{i\text{th Bernoulli trial is a success}\},i=1,2,\dots$, as in the previous section. Let's consider a particular sequence of outcomes such that there are $r$ successes in $n$ trials: $$\underset{r\text{ successes}}{\underbrace{S\cdots S}}\stackrel{n-r\text{ failures}}{\overbrace{F\cdots F}}$$ Its probability is $$\mathbb{P}(S_1\cap \cdots S_r\cap S_{r+1}^c\cap \cdots \cap S_n^c)\stackrel{\text{ indpt. }}{=}\mathbb{P}(S_1)\cdots \mathbb{P}(S_r)\mathbb{P}(S_{r+1}^c)\cdots \mathbb{P}(S_n^c)=p^r(1-p{)}^{n-r}$$ Since the probability of other sequences with some of $r$ successes occurring in other trials is the Template:Colored em, and there are $(\genfrac{}{}{0ex}{}{n}{r})$ distinct possible sequences, $$\mathbb{P}(\{r\text{ successes in }n\text{ trials}\})=(\genfrac{}{}{0ex}{}{n}{r})p^r(1-p{)}^{n-r}.$$ This is the pmf of a random variable following the Template:Colored em.

Bernoulli distribution

Bernoulli distribution is simply a special case of Template:Colored em distribution, as follows: Template:Colored definition Template:Colored remark

Poisson distribution

Motivation

The Poisson distribution can be viewed as the 'limit case' for the binomial distribution.

Consider $n$ independent Bernoulli trials with success probability $p=\lambda /n$. By the binomial distribution, $$\mathbb{P}(r\text{ successes in }n\text{ trials})=(\genfrac{}{}{0ex}{}{n}{r})(\lambda /n{)}^r(1-\lambda /n{)}^{n-r}.$$

After that, consider an unit time interval, with (positive) Template:Colored em $\lambda$ of a rare event (i.e. the Template:Colored em of number of occurrence of the rare event is $\lambda$). We can divide the unit time interval to $n$ time subintervals of time length $1/n$ each. If $n$ is Template:Colored em and $p$ is Template:Colored em, such that the probability for occurrence of two or more Template:Colored em at a single time interval is negligible, then the probability for occurrence of Template:Colored em for each time subinterval is $p=\lambda /n$ by definition of mean. Then, we can view the unit time interval as a sequence of $n$ Bernoulli trials with success probability $p=\lambda /n$. After that, we can use $\mathrm{Binom}(n,\lambda /n)$ to model the number of occurrences of Template:Colored em. To be more precise, $$\begin{array}{cc}\mathbb{P}(\underset{r\text{ rare events in the unit time}}{\underbrace{r\text{ successes in }n\text{ trials}}})& =(\genfrac{}{}{0ex}{}{n}{r})(\lambda /n{)}^r(1-\lambda /n{)}^{n-r}\\ & =\frac{n(n-1)\cdots (n-r+1)}{r!}({\lambda }^r/n^r)(1-\lambda /n{)}^{n-r}\\ & =({\lambda }^r/r!)\stackrel{\to 1\text{ as }n\to \mathrm{\infty }}{\overbrace{(1-\underset{\to 0\text{ as }n\to \mathrm{\infty }}{\underbrace{1/n}})\cdots (1-\underset{\to 0\text{ as }n\to \mathrm{\infty }}{\underbrace{(r-1)/n}})}}\underset{\to e^{-\lambda }\text{ as }n\to \mathrm{\infty }}{\underbrace{(1-\lambda /n{)}^{\stackrel{\to n\text{ as }n\to \mathrm{\infty }}{\overbrace{n-r}}}}}\\ & \to e^{-\lambda }{\lambda }^r/r!\text{ as }n\to \mathrm{\infty }.\end{array}$$ This is the pmf of a random variable following the Template:Colored em, and this result is known as the Template:Colored em (or law of rare events). We will introduce it formally after introducing the definition of Template:Colored em.

Definition

Proof. The result follows from the result proved above: the pmf of $\mathrm{Binom}(n,\lambda /n)$ approaches the pmf of $\mathrm{Pois}(\lambda )$ as $n\to \mathrm{\infty }$.

$◻$

Geometric distribution

Motivation

Consider a sequence of independent Bernoulli trials with success probability $p$. We would like to calculate the probability $\mathbb{P}(\{x\text{ failures before first success}\})$. By considering this sequence of outcomes: $$\underset{x\text{ failures}}{\underbrace{F\cdots F}}S,$$ we can calculate that $$\mathbb{P}(\{x\text{ failures before first success}\})=(1-p{)}^{x}p,x\in \mathrm{supp}(X)=\{0,1,2,\dots \}$$ This is the pmf of a random variable following the Template:Colored em.

Definition

Proof. $$\begin{array}{cc}\mathbb{P}(X>m+n|X\ge m)& \stackrel{\text{ def }}{=}\frac{\mathbb{P}(\stackrel{=X>m+n}{\overbrace{X>m+n\cap X\ge m)}}}{\mathbb{P}(X\ge m)}\\ & \stackrel{\text{ def }}{=}\frac{\cancel{p}((1-p{)}^{m+n+1}+(1-p{)}^{m+n+2}+\cdots )}{\cancel{p}((1-p{)}^m+(1-p{)}^{m+1}+\cdots )}\\ & =\frac{(1-p{)}^{\cancel{m}+n+1}\cancel{/(1-(1-p))}}{\cancel{(1-p{)}^m}\cancel{/(1-(1-p))}}& \text{by geometric series formula}\\ & =(1-p{)}^{n+1}\cdot \frac{p}{p}\\ & =p\cdot \frac{(1-p{)}^{n+1}}{1-(1-p)}\\ & =p((1-p{)}^{n+1}+(1-p{)}^{n+2}+\cdots )& \text{by geometric series formula}\\ & \stackrel{\text{ def }}{=}\mathbb{P}(X>n)& \text{since }X>n\iff X=n+1,n+2,\dots .\\ \end{array}$$

• In particular, $X>m+n\cap X\ge m=X>m+n$ since $\underset{X=m+n+1,m+n+2,\dots }{\underbrace{X>m+n}}⊊\underset{X=m,m+1,\dots }{\underbrace{X\ge m}}$.

$◻$

Negative binomial distribution

Motivation

Consider a sequence of independent Bernoulli trials with success probability $p$. We would like to calculate the probability $\mathbb{P}(\{x\text{ failures before }k\text{th success}\})$. By considering this sequence of outcomes: $$\stackrel{x+k-1\text{ trials}}{\overbrace{\underset{x_1\text{ failures}}{\underbrace{F\cdots F}}S\underset{x_2\text{ failures}}{\underbrace{F\cdots F}}S\cdots \underset{x_k\text{ failures}}{\underbrace{F\cdots F}}}}\stackrel{k\text{th success}}{\overbrace{S}},x_1+x_2+\cdots +x_k=x,$$ we can calculate that $$\mathbb{P}(\{x\text{ failures before }k\text{th success}\})=(1-p{)}^x{p}^k,x\in \mathrm{supp}(X)=\{0,1,2,\dots \}.$$ Since the probability of other sequences with some of $x$ failures occurring in other trials (and some of $k-1$ successes (excluding the $k$th success, which must occur in the last trial) occurring in other trials), is the Template:Colored em, and there are $(\genfrac{}{}{0ex}{}{x+k-1}{x})$ (or $(\genfrac{}{}{0ex}{}{x+k-1}{k-1})$, which is the same numerically) distinct possible sequences , $$\mathbb{P}(\{x\text{ failures before }k\text{th success}\})=(\genfrac{}{}{0ex}{}{x+k-1}{x})(1-p{)}^x{p}^k,x\in \mathrm{supp}(X)=\{0,1,2,\dots \}.$$ This is the pmf of a random variable following the Template:Colored em.

Hypergeometric distribution

Motivation

Consider a sample of size $n$ are drawn without replacement from a population size $N$, containing $K$ objects of type 1 and $N-K$ of another type. Then, the probability $$\mathbb{P}(\{k\text{ type 1 objects are found when }n\text{ objects are drawn from }N\text{ objects}\})=\underset{\text{type 1}}{\underbrace{(\genfrac{}{}{0ex}{}{K}{k})}}\stackrel{\text{another type}}{\overbrace{(\genfrac{}{}{0ex}{}{N-K}{n-k})}}/\underset{\text{all outcomes}}{\underbrace{(\genfrac{}{}{0ex}{}{N}{n})}},k\in \{\max \{n-N+K,0\},\dots ,\min \{K,n\}\}$$ .

• $(\genfrac{}{}{0ex}{}{K}{k})$: unordered selection of $k$ objects of type 1 from $K$ (distinguishable) objects of type 1 without replacement;
• $(\genfrac{}{}{0ex}{}{N-K}{n-k})$: unordered selection of $n-k$ objects of another type from $N-K$ (distinguishable) objects of another type without replacement;
• $(\genfrac{}{}{0ex}{}{N}{n})$: unordered selection of $n$ objects from $N$ (distinguishable) objects without replacement.

This is the pmf of a random variable following the Template:Colored em.

Finite discrete distribution

This type of distribution is a generalization of all discrete distribution with finite support, e.g. Bernoulli distribution and hypergeometric distribution.

Another special case of this type of distribution is Template:Colored em, which is similar to the Template:Colored em (will be discussed later). Template:Colored definition Template:Colored remark Template:Colored definition Template:Colored remark Template:Colored example Template:Colored example

Distributions of a continuous random variable

Uniform distribution (continuous)

The Template:Colored em is a model for 'no preference', i.e. all intervals of the same length on its support are Template:Colored em (it can be seen from the pdf corresponding to continuous uniform distribution). There is also Template:Colored em uniform distribution, but it is less important than Template:Colored em uniform distribution. So, from now on, simply 'uniform distribution' refers to the Template:Colored em one, instead of the discrete one. Template:Colored definition Template:Colored remark Template:Colored proposition

Proof. $$F(x)=\underset{-\mathrm{\infty }}{\overset{x}{\int }}\frac{\mathrm{𝟏}\{a\le x\le b\}}{b-a}dy=\frac{1}{b-a}\underset{a}{\overset{x}{\int }}\mathrm{𝟏}\{a\le x\le b\}dy=\{\begin{array}{cc}0/(b-a),& x<a;\\ [][y{]}_a^x/(b-a),& a\le x\le b;\\ [][y{]}_a^b/(b-a),& x>b.\end{array}$$ Then, the result follows.

$◻$

Exponential distribution

The Template:Colored em distribution with Template:Colored em parameter $\lambda$ is often used to describe the Template:Colored em of rare events with rate $\lambda$.

Comparing this with the Template:Colored em distribution, the Template:Colored em distribution describes the interarrival Template:Colored em of rare events, while Template:Colored em distribution describes the Template:Colored em of occurrences of rare events within a fixed time interval.

By definition of Template:Colored em, when the Template:Colored em $\uparrow$, then Template:Colored em $\downarrow$ (i.e. frequency of the rare event $\uparrow$).

So, we would like the pdf to be more skewed to left when $\lambda \uparrow$(i.e. the pdf has higher value for small $x$ when $\lambda \uparrow$), so that areas under the pdf for intervals involving small value of $x$ $\uparrow$ when $\lambda \uparrow$.

Also, since with a fixed rate $\lambda$, the interarrival time should be less likely of higher value. So, intuitively, we would also like the pdf to be a strictly Template:Colored em function, so that the probability involved (area under the pdf for some interval) $\downarrow$ when $x\uparrow$.

As we can see, the pdf of exponential distribution satisfies both of these properties. Template:Colored definition Template:Colored proposition

Proof. Suppose $X\sim \mathrm{Exp}(\lambda )$. The cdf of $X$ is $$\begin{array}{cc}F(x)& =\underset{-\mathrm{\infty }}{\overset{x}{\int }}\lambda e^{-\lambda y}\mathrm{𝟏}\{y\ge 0\}dy\\ & =\{\begin{array}{cc}\underset{0}{\overset{x}{\int }}\lambda e^{-\lambda y}dy,& x\ge 0;\\ 0,& x<0\end{array}& (\text{When }x<0,x\notin \mathrm{supp}(X),\text{ so }F(x)=\mathbb{P}(X\le x)=0)\\ & =\mathrm{𝟏}\{x\ge 0\}\lambda \underset{0}{\overset{x}{\int }}{e}^{-\lambda y}dy\\ & =\mathrm{𝟏}\{x\ge 0\}\frac{\lambda }{-\lambda }[e^{-\lambda }y{]}_0^x\\ & =-\mathrm{𝟏}\{x\ge 0\}(e^{-\lambda x}-1)\\ & =(1-e^{-\lambda x})\mathrm{𝟏}\{x\ge 0\}.\\ \end{array}$$

$◻$

Proof. $$\mathbb{P}(X>s+t|X>s)\stackrel{\text{ def }}{=}\frac{\mathbb{P}(X>s+t\cap X>s)}{\mathbb{P}(X>s)}=\frac{\mathbb{P}(X>s+t)}{\mathbb{P}(X>s)}=\frac{1-(1-e^{-\lambda (s+t)})}{1-(1-e^{-\lambda s})}=\frac{e^{-\lambda (s+t)}}{e^{-\lambda s}}=e^{-\lambda t}=\mathbb{P}(X>t).$$

$◻$

Gamma distribution

Template:Colored em distribution is a generalized Template:Colored em distribution, in the sense that we can also change the Template:Colored em of the pdf of Template:Colored em distribution. Template:Colored definition Template:Colored remark

Beta distribution

Template:Colored em distribution is a generalized $𝒰[0,1]$, in the sense that we can also change the Template:Colored em of the pdf, using Template:Colored em. Template:Colored definition Template:Colored remark

Cauchy distribution

The Template:Colored em distribution is a Template:Colored em distribution . As a result, it is a 'pathological' distribution, in the sense that it has some counter-intuitive properties, e.g. undefined mean and variance, despite its mean and variance Template:Colored em to be defined when we look at its graph directly. Template:Colored definition Template:Colored remark

Normal distribution (very important)

The normal or Gaussian distribution is a thing of beauty, appearing in many places in nature. This is probably because sample means or sample sums often follow Template:Colored em distributions Template:Colored em by Template:Colored em. As a result, the Template:Colored em distribution is important in statistics.

Proof. Assume $a>0$ . Let $F_X$ and $F_Y$ be cdf of $X$ and $Y$ respectively. Since $$F_Y(y)=\mathbb{P}(Y\le y)=\mathbb{P}(aX+b\le y)=\mathbb{P}(X\le (y-b)/a)=F_X((y-b)/a),$$ by differentiation, $$\begin{array}{cc}{f}_Y(y)& =\frac{1}{a}{f}_X((y-b)/a)\\ & =\frac{1}{a\sqrt{2\pi {\sigma }^2}}\exp (-((y-b)/a-\mu {)}^2/2{\sigma }^2)\\ & =\frac{1}{\sqrt{2\pi a^2{\sigma }^2}}\exp (-(y-(a\mu +b){)}^2/2a^2{\sigma }^2)& \text{since }a>0,\\ \end{array}$$ which is the pdf of $𝒩(a\mu +b,a^2{\sigma }^2)$.

$◻$

Important distributions for statistics especially

The following distributions are important in statistics especially, and they are all related to normal distribution. We will introduce them briefly.

Chi-squared distribution

The Template:Colored em distribution is a special case of Gamma distribution, and also related to Template:Colored em distribution.

Student's t-distribution

The Template:Colored em is related to Template:Colored em distribution and Template:Colored em distribution. Template:Colored definition Template:Colored remark

F-distribution

The $F$-distribution is sort of a generalized Student's $t$-distribution, in the sense that it has one more changeable parameter for another degrees of freedom. Template:Colored definition Template:Colored remark If you are interested in knowing how Template:Colored em, Template:Colored em, and Template:Colored em are useful in statistics, then you may briefly look at, for instance, Statistics/Interval Estimation (applications in confidence interval construction) and Statistics/Hypothesis Testing (applications in hypothesis testing).

Joint distributions

Multinomial distribution

Motivation

Multinomial distribution is Template:Colored em binomial distribution, in the sense that each trial has more than two outcomes.

Suppose $n$ objects are to be allocated to $k$ cells independently, for which each object is allocated to Template:Colored em cell, with probability $p_i$ to be allocated to the $i$th cell ($i=1,2,\dots ,k$) . Let $X_i$ be the number of objects allocated to cell $i$. We would like to calculate the probability $\mathbb{P}(𝐗\stackrel{\text{ def }}{=}(X_1,\dots ,X_k{)}^T=𝐱\stackrel{\text{ def }}{=}(x_1,\dots ,x_k{)}^T)$, i.e. the probability that $i$th cell has $x_i$ objects.

We can regard each allocation as an independent trial with $k$ outcomes (since it can be allocated to one and only one of $k$ cells). We can recognize that the allocation of $n$ objects is partition of $n$ objects into $k$ groups. There are hence $(\genfrac{}{}{0ex}{}{n}{x_1,\dots ,x_k})$ ways of allocation.

So, $\mathbb{P}(𝐗=𝐱)=(\genfrac{}{}{0ex}{}{n}{x_1,\dots ,x_k})p_1^{x_1}\cdots p_k^{x_k}.$ In particular, the probability of allocating $x_i$ objects to $i$th cell is $p_i^{x_i}$ by independence, and so that of a particular case of allocation of $n$ objects to $k$ cells is $p_1^{x_1}\cdots p_k^{x_k}$ by independence.

Multivariate normal distribution

Template:Colored em normal distribution is, as suggested by its name, a multivariate (and also generalized) version of the normal distribution (univariate). Template:Colored definition Template:Colored remark Template:Info Template:Colored proposition

Proof. For the bivariate normal distribution,

• the Template:Colored em is $\mathit{\mu }=({\mu }_X,{\mu }_Y)$;
• the Template:Colored em is $\mathrm{\Sigma }=\left(\begin{array}{cc}\mathrm{Cov}(X,X)& \mathrm{Cov}(X,Y)\\ \mathrm{Cov}(Y,X)& \mathrm{Cov}(Y,Y)\end{array}\right)=\left(\begin{array}{cc}\mathrm{Var}(X)& \mathrm{Cov}(X,Y)\\ \mathrm{Cov}(X,Y)& \mathrm{Var}(Y)\end{array}\right)=\left(\begin{array}{cc}{\sigma }_X^2& \rho {\sigma }_X{\sigma }_Y\\ \rho {\sigma }_X{\sigma }_Y& {\sigma }_Y^2\end{array}\right).$
• Hence,

$$\begin{array}{cc}(𝐱-\mathit{\mu }{)}^T{\mathrm{\Sigma }}^{-1}(𝐱-\mathit{\mu })& =\frac{1}{\det \mathrm{\Sigma }}{((x-{\mu }_X,y-{\mu }_Y{)}^T)}^T\left(\begin{array}{cc}{\sigma }_Y^2& -\rho {\sigma }_X{\sigma }_Y\\ -\rho {\sigma }_X{\sigma }_Y& {\sigma }_X^2\end{array}\right)(x-{\mu }_X,y-{\mu }_Y{)}^T)\\ & =\frac{1}{\det \mathrm{\Sigma }}\left(\begin{array}{cc}x-{\mu }_X& y-{\mu }_Y\end{array}\right)\left(\begin{array}{cc}{\sigma }_Y^2& -\rho {\sigma }_X{\sigma }_Y\\ -\rho {\sigma }_X{\sigma }_Y& {\sigma }_X^2\end{array}\right)\left(\begin{array}{c}x-{\mu }_X\\ y-{\mu }_Y\end{array}\right)\\ & =\frac{1}{\det \mathrm{\Sigma }}\left(\begin{array}{cc}(x-{\mu }_X){\sigma }_Y^2-(y-{\mu }_Y)\rho {\sigma }_X{\sigma }_Y& -(x-{\mu }_X)\rho {\sigma }_X{\sigma }_Y+(y-{\mu }_Y){\sigma }_X^2\end{array}\right)\left(\begin{array}{c}x-{\mu }_X\\ y-{\mu }_Y\end{array}\right)\\ & =\frac{1}{\underset{{\sigma }_X^2{\sigma }_Y^2-(\rho {\sigma }_X{\sigma }_Y{)}^2}{\underbrace{\det \mathrm{\Sigma }}}}((x-{\mu }_X{)}^2{\sigma }_Y^2\underset{=-2\rho (x-{\mu }_X)(y-{\mu }_Y){\sigma }_X{\sigma }_Y}{\underbrace{-(x-{\mu }_X)(y-{\mu }_Y)\rho {\sigma }_X{\sigma }_Y-(x-{\mu }_X)(y-{\mu }_Y)\rho {\sigma }_X{\sigma }_Y}}+(y-{\mu }_Y{)}^2{\sigma }_X^2)\\ & =\frac{(x-{\mu }_X{)}^2{\sigma }_Y^2-2\rho (x-{\mu }_X)(y-{\mu }_Y){\sigma }_X{\sigma }_Y+(y-{\mu }_Y{)}^2{\sigma }_X^2}{{\sigma }_X^2{\sigma }_Y^2(1-\rho {)}^2}\\ & =\frac{1}{1-{\rho }^2}({\left(\frac{x-{\mu }_X}{{\sigma }_X}\right)}^2-2\rho \left(\frac{(x-{\mu }_X)(y-{\mu }_Y)}{{\sigma }_X{\sigma }_Y}\right)+{\left(\frac{y-{\mu }_Y}{{\sigma }_Y}\right)}^2).\end{array}$$

• It follows that the joint pdf is

$$\begin{array}{cc}f(x,y)& =\frac{1}{\sqrt{(2\pi {)}^2\det \mathrm{\Sigma }}}\exp (-\frac{1}{2}\cdot \frac{1}{1-{\rho }^2}({\left(\frac{x-{\mu }_X}{{\sigma }_X}\right)}^2-2\rho \left(\frac{(x-{\mu }_X)(y-{\mu }_Y)}{{\sigma }_X{\sigma }_Y}\right)+{\left(\frac{y-{\mu }_Y}{{\sigma }_Y}\right)}^2))\\ & =\frac{1}{2\pi \sqrt{{\sigma }_X^2{\sigma }_Y^2(1-{\rho }^2)}}\exp \left(\frac{-1}{2(1-{\rho }^2)}({\left(\frac{x-{\mu }_X}{{\sigma }_X}\right)}^2-2\rho \left(\frac{(x-{\mu }_X)(y-{\mu }_Y)}{{\sigma }_X{\sigma }_Y}\right)+{\left(\frac{y-{\mu }_Y}{{\sigma }_Y}\right)}^2)\right)\\ & =\frac{1}{2\pi {\sigma }_X{\sigma }_Y\sqrt{1-{\rho }^2}}\exp \left(\frac{-1}{2(1-{\rho }^2)}({\left(\frac{x-{\mu }_X}{{\sigma }_X}\right)}^2-2\rho \left(\frac{x-{\mu }_X}{{\sigma }_X}\right)\left(\frac{y-{\mu }_Y}{{\sigma }_Y}\right)+{\left(\frac{y-{\mu }_Y}{{\sigma }_Y}\right)}^2)\right).\\ \end{array}$$

$◻$