View source for Probability/Joint Distributions and Independence

<noinclude>
{{nav}}
</noinclude>
{{TOC}}
{{clear}}
== Motivation ==
Suppose we are given a pmf of a discrete random variable <math>X</math> and a pmf of a discrete random variable <math>Y</math>. For example,
<math display="block">
f_X(x)=(\mathbf 1\{x=0\}+\mathbf 1\{x=1\})/2\quad\text{and}\quad f_Y(y)=(\mathbf 1\{y=0\}+\mathbf 1\{y=2\})/2
</math>
We cannot tell the relationship between <math>X</math> and <math>Y</math> with only such information. They may be related or not related.

For example, the random variable <math>X</math> may be defined as 
<math>X=1</math> if head comes up and <math>X=0</math> otherwise from tossing a fair coin, 
and the random variable <math>Y</math> may be defined as <math>Y=2</math> 
if head comes up and <math>Y=0</math> otherwise from tossing the coin another time.
In this case, <math>X</math> and <math>Y</math> are unrelated.

Another possibility is that the random variable <math>Y</math> is defined as
<math>Y=2X</math> if head comes up in the first coin tossing, and <math>Y=0</math> otherwise.
In this case, <math>X</math> and <math>Y</math> are related.

Yet, in the above two examples, the pmf of <math>X</math> and <math>Y</math> are exactly the same.

Therefore, to tell the {{colored em|relationship}} between <math>X</math> and <math>Y</math>, we define the {{colored em|joint}} cumulative distribution function, or joint cdf. 

== Joint distributions ==
{{colored definition|(Joint cumulative distribution function)
Let <math>X_1,\dotsc,X_n</math> be random variables defined on a sample space <math>\Omega</math>. The {{colored em|joint}} cumulative distribution function (cdf) of random variables <math>X_1,\dotsc,X_n</math> is
<math display=block>
F(x_1,\dotsc,x_n)=\mathbb P(X_1\le x_1\cap\cdots\cap X_n\le x_n)
=\mathbb P\left(\bigcap_{i=1}^n\{\omega\in\Omega:X_i(\omega)\le x_i\}\right).
</math>}}
Sometimes, we may want to know the random behaviour in one of the random variables involved in a joint cdf.
We can do this by computing the marginal cdf from joint cdf. The definition of marginal cdf is as follows:
{{colored definition|(Marginal cumulative distribution function)
The cumulative distribution function <math>F_{X_i}</math> of each random variable <math>X_i</math> is {{colored em|marginal cumulative distribution function}} (cdf) of <math>X_i</math> which is a member in the <math>n</math> random variables <math>X_1,\dotsc,X_n</math>.}}
{{colored remark|Actually, the marginal cdf of <math>X_i</math> is simply the cdf of <math>X_i</math> (which is in one variable). We have already discussed this kind of cdf in previous chapters.}}
{{colored proposition|(Obtaining marginal cdf from joint cdf)
Given a joint cdf <math>F(x_1,\dotsc,x_n)</math>, the marginal cdf of <math>X_i</math> is
<math display=block>
F_{X_i}(x)=F(\infty,\dotsc,\infty,\underbrace{x}_{i\text{-th position}},\infty,\dotsc,\infty).
</math>
}}
{{colored proof|When we set the arguments other than <math>i</math>-th argument to be <math>\infty</math>, e.g. <math>X_1\le\infty\Leftrightarrow\lim_{x\to\infty}X_1\le x</math> , the joint cdf becomes
<math display=block>
\begin{align}
\mathbb P(X_1\le\infty\cap\cdots\cap X_{i-1}\le\infty\cap X_i\le x\cap X_{i+1}\le\infty\cap\cdots\cap X_n\le\infty)
&=\underbrace{\mathbb P(X_1\le\infty\cap\cdots\cap X_{i-1}\le\infty)}_1\mathbb P(X_i\le x)\underbrace{\mathbb P(X_{i+1}\le\infty\cap\cdots\cap X_n\le\infty)}_1\qquad\text{by independence}\\
&=\mathbb P(X_i\le x)\\
&=F_{X_i}(x)
\end{align}
</math>}}
{{colored remark|In general, we cannot deduce the joint cdf from a given set of marginal cdf's.}}
{{colored example|Consider the joint cdf of random variables <math>X</math> and <math>Y</math>: 
<math display=block>
F(x,y)=1-e^{-x}-e^{-y}+e^{-xy}.
</math>
The marginal cdf of <math>Y</math> is
<math display=block>
F_Y(y)=\lim_{x\to\infty}(1-e^{-x}-e^{-y}+e^{-xy})=1-e^{-y}.
</math>}}
Similar to the one-variable case, we have joint pmf and joint pdf. Also, analogously, we have marginal pmf and marginal pdf.

{{colored definition|(Joint probability mass function)
The {{colored em|joint probability mass function}} (joint pmf) of <math>X_1,\dotsc,X_n</math> is
<math display=block>
f(x_1,\dotsc,x_n)=\mathbb P\big((X_1,\dotsc,X_n)=(x_1,\dotsc,x_n)\big),\quad (x_1,\dotsc,x_n)\in\mathbb R^n.
</math>
}}
{{colored definition|(Marginal probability mass function)
The {{colored em|marginal probability mass function}} (marginal pmf) of each <math>X_i</math> which is a member of the <math>n</math> random variables <math>X_1,X_2,\dotsc,X_n</math> is
<math display=block>
f_{X_i}(x)=\mathbb P(X_i=x),\quad x\in\mathbb R.
</math>}}
{{colored proposition|(Obtaining marginal pmf from joint pmf)
For discrete random variables <math>X_1,\dotsc,X_n</math> with joint pmf <math>f</math>, the marginal pmf of <math>X_i</math> is
<math display=block>
f_{X_i}({\color{red}x})=\underbrace{\sum_{u_1}\cdots\sum_{u_{i-1}}\sum_{u_{i+1}}\cdots\sum_{u_n}}_{n-1\;\text{summations}}f(u_1,\dotsc,u_{i-1},{\color{red}x},u_{i+1},\dotsc,u_n).
</math>}}
{{colored proof|Consider the case in which there are only two random variables, say <math>X</math> and <math>Y</math>.
Then, we have
<math display=block>
\sum_{\color{green}y} f({\color{red}x},{\color{green}y})=\sum_{\color{green}y}\mathbb P(X={\color{red}x}\cap {\color{green}Y=y})
=\mathbb P(X={\color{red}x})\qquad\text{by law of total probability}.
</math>
Similarly, in general case, we have
<math display=block>
\begin{align}
\sum_{\color{green}u_n}f(u_1,\dotsc,u_{i-1},{\color{red}x},u_{i+1},\dotsc,{\color{green}u_n})
&=\sum_{\color{green}u_n}\mathbb P(X_1\le u_1\cap\cdots\cap X_{i-1}u_{i-1}\cap X_i\le{\color{red}x}\cap X_{i+1}\le u_{i+1}\cap\cdots\cap X_{n-1}\le u_{n-1}\cap{\color{green}X_n\le u_n})\\
&=\mathbb P(X_1\le u_1\cap\cdots\cap X_{i-1}u_{i-1}\cap X_i\le{\color{red}x}\cap X_{i+1}\le u_{i+1}\cap\cdots\cap X_{n-1}\le u_{n-1})\qquad\text{by law of total probability}.
\end{align}
</math>
Then, we perform similar process on each of the other variables (<math>n-2</math> left), with one extra summation sign added for each process.
Thus, in total we will have <math>n-1</math> summation sign, and we will finally get the desired result.}}
{{colored remark|This process may sometimes be called 'summing over each possible value of other variables'.}}
{{colored example|
Suppose we throw a fair six-faced dice two times. Let <math>X</math> be the number facing up in the first throw, and <math>Y</math> be the number facing up in the second throw.
Then, the {{colored em|joint pmf}} of <math>(X,Y)</math> is
<math display=block>
f(x,y)=\mathbb P(X=x\cap Y=y)=\frac{1}{6}\cdot\frac{1}{6}=\frac{1}{36}.
</math>
in which <math>x,y\in\{1,2,3,4,5,6\}</math>, and <math>f(x,y)=0</math> 
otherwise.
Also, the {{colored em|marginal pmf}} of <math>X</math> is
<math display=block>
f_X(x)=\sum_{y}f(x,y)=f(x,1)+f(x,2)+\cdots+f(x,6)=6(1/36)=\frac{1}{6}
</math>
in which <math>x\in\{1,2,3,4,5,6\}</math>, and <math>f_X(x)=0</math> otherwise.

By symmetry (replace all <math>X</math> with <math>Y</math> and replace all <math>x</math> with <math>y</math>), the {{colored em|marginal pmf}} of <math>Y</math> is
<math display=block>
f_Y(y)=\frac{1}{6}
</math>
in which <math>y\in\{1,2,3,4,5,6\}</math>, and <math>f_Y(x)=0</math> otherwise.
}}
{{colored exercise|
Suppose there are two red balls and one blue ball in a box, and we draw two balls one by one from the box with replacement.
Let <math>X=1</math> if the ball from the first draw is red, and <math>X=0</math> otherwise.
Let <math>Y=1</math> if the ball from the second draw is red, and <math>Y=0</math> otherwise.
<quiz display=simple>
{Calculate the marginal pmf of <math>X</math>. 
|type="()"}
+ <math>f_X(x)=(\mathbf 1\{x=0\}+2\cdot\mathbf 1\{x=1\})/3</math> 
||<math>\mathbb P(X=1)=2/3</math> and <math>\mathbb P(X=0)=1/3</math>. 
- <math>f_X(x)=(\mathbf 1\{x=1\}+2\cdot\mathbf 1\{x=0\})/3</math> 
- <math>f_X(x)=2/3</math> 
- <math>f_X(x)=(\mathbf 1\{x=1\}+\mathbf 1\{x=0\})/2</math> 

{Calculate the joint pmf of <math>(X,Y)</math>.
|type="()"}
+ <math>f(x,y)=(1/9)(\mathbf 1\{(x,y)=(0,0)\}+2\cdot\mathbf 1\{(x,y)=(0,1)\}+2\cdot\mathbf 1\{(x,y)=(1,0)\}+4\cdot\mathbf 1\{(x,y)=(1,1)\}</math>)
- <math>f(x,y)=(1/9)(4\cdot\mathbf 1\{(x,y)=(0,0)\}+2\cdot\mathbf 1\{(x,y)=(0,1)\}+2\cdot\mathbf 1\{(x,y)=(1,0)\}+\mathbf 1\{(x,y)=(1,1)\}</math>) 
- <math>f(x,y)=(2/9)(\mathbf 1\{(x,y)=(0,0)+\mathbf 1\{(x,y)=(0,1)\}+\mathbf 1\{(x,y)=(1,0)\}+\mathbf 1\{(x,y)=(1,1)\}</math>) 
</quiz>
}}
{{colored exercise|
Recall the example in the motivation section.

(a) Suppose we toss a fair coin twice. Let <math>X=\mathbf 1\{\text{head comes up}\}</math> and <math>Y=2\cdot\mathbf 1\{\text{head comes up}\}</math>. 
Show that joint pmf of <math>(X,Y)</math> is
<math display=block>
f(x,y)=\frac{\mathbf 1\{x\in\{0,1\}\cap y\in\{0,2\}\}}{4}.
</math>

(b) Suppose we toss a fair coin once. Let <math>X=\mathbf 1\{\text{head comes up}\}</math> and <math>Y=2X</math>. 
Show that joint pmf of <math>(X,Y)</math> is
<math display=block>
f(x,y)=\frac{\mathbf 1\{x\in\{0,1\}\cap y=2x\}}{2}.
</math>

(c) Show that marginal pmf of <math>X</math> and <math>Y</math> are
<math display=block>
f_X(x)=\frac{\mathbf 1\{x\in\{0,1\}\}}{2}\quad\text{and}\quad f_Y(y)=\frac{\mathbf 1\{y\in\{0,2\}\}}{2}
</math>
in each of the situations in (a) and (b).
(Hint: for part (b), we need to put value in the variable in the indicator)
}}
{{hide|Proof.|
(a) Since the support of <math>(X,Y)</math> is <math>\{(0,0),(0,2),(1,0),(1,2)\}</math>, 
the joint pmf of <math>(X,Y)</math> is
<math display=block>
f(x,y)=\mathbb P(X=0\cap Y=0)+\mathbb P(X=0\cap Y=2)+\mathbb P(X=1\cap Y=0)+\mathbb P(X=1\cap Y=2)=\frac{\mathbf 1\{x\in\{0,1\}\cap y\in\{0,2\}\}}{4}.
</math>

(b) Since the support of <math>(X,Y)</math> is <math>x\in\{0,1\}\cap y=2x</math>,
the joint pmf of <math>(X,Y)</math> is
<math display=block>
f(x,y)=\mathbb P(X=0\cap Y=2(0)=0)+\mathbb P(X=1\cap Y=2(1)=2)=\frac{\mathbf 1\{x\in\{0,1\}\cap y=2x\}}{2}.
</math>

(c) Part (a): marginal pmf of <math>X</math> is
<math display=block>
f_X(x)=f(x,0)+f(x,2)=\frac{2\mathbf 1\{x\in\{0,1\}\}}{4}=\frac{\mathbf 1\{x\in\{0,1\}\}}{2},
</math>
and marginal pmf of <math>Y</math> is
<math display=block>
f_Y(y)=f(0,y)+f(1,y)=\frac{2\mathbf 1\{y\in\{0,2\}\}}{4}=\frac{\mathbf 1\{y\in\{0,2\}\}}{2}.
</math>

Part (b): 
The marginal pmf of <math>X</math> is
<math display=block>
f_X(x)=f(x,0)+f(x,2)=\frac{\mathbf 1\{\overbrace{x\in\{0,1\}\cap 0=2x}^{x=0}\}}{2}+\frac{\mathbf 1\{\overbrace{x\in\{0,1\}\cap 2=2x}^{x=1}\}}{2}
=\frac{\overbrace{\mathbf 1\{x=0\}+\mathbf 1\{x=1\}}^{\mathbf 1\{x=0\cup x=1\}}}{2}=\frac{\mathbf 1\{x\in\{0,1\}\}}{2}.
</math>
Similarly, the marginal pmf of <math>Y</math> is
<math display=block>
f_Y(y)=f(0,y)+f(1,y)=\frac{\mathbf 1\{0\in\{0,1\}\cap y=0\}}{2}+\frac{\mathbf 1\{1\in\{0,1\}\cap y=2\}}{2}=\frac{\overbrace{\mathbf 1\{0\in\{0,1\}\}}^1\mathbf 1\{y=0\}+\overbrace{\mathbf 1\{1\in\{0,1\}\}}^1\mathbf 1\{y=2\}}{2}=\frac{\mathbf 1\{y\in\{0,2\}\}}{2}.
</math>
}}
For {{colored em|jointly}} continuous random variables, the definition is generalized version of the one for continuous random variables (univariate case).
{{colored definition|(Jointly continuous random variable)
Random variables <math>X_1,\dotsc,X_n</math> are {{colored em|jointly continuous}} if
<math display=block>\mathbb P\big((X_1,\dotsc,X_n)\in S\big)=\int\dotsi\int_S f(x_1,\dotsc,x_n)\,dx_1\cdots\,dx_n,\quad S\subseteq\mathbb R^n,</math>
for some nonnegative function <math>f</math>. 
}}
{{colored remark|
* The function <math>f</math> is the {{colored em|joint probability density function}} (joint pdf) of <math>X_1,\dotsc,X_n</math>.
* Similarly, <math>f(x_1,\dotsc,x_n)\,dx_1\cdots\,dx_n</math> can be interpreted as the probability over the 'infinitesimal' region <math>[x_1,x_1+dx_1]\times\dotsb\times[x_n,x_n+dx_n]</math>, and <math>f(x_1,\dotsc,x_n)</math> can be interpreted as the density of the probability over that 'infinitesimal' region, i.e. <math>\frac{\mathbb P\big(X\in[x_1,x_1+dx_1]\times\dotsb\times[x_n,x_n+dx_n]\big)}{dx_1\dotsb dx_n}</math>, intuitively and non-rigourously. 
* By setting <math>S=(-\infty,x_1]\times\dotsb\times(-\infty,x_n]</math>, the cdf
<math display=block>
F(x_1,\dotsc,x_n)=\underbrace{\int_{-\infty}^{x_1}\cdots\int_{-\infty}^{x_n}}_{n\;\text{integrations}}f(u_1,\dotsc,u_n)\,du_n\cdots\,du_1,
</math>
: which is similar to the univariate case.
}}
{{colored definition|(Marginal probability density function)
The pdf <math>f_{X_i}</math> of each <math>X_i</math> which is a member of the <math>n</math> random variables <math>X_1,X_2,\dotsc,X_n</math> 
is the {{colored em|marginal probability density function}} (marginal pdf) of <math>X_i</math>.}}
{{colored proposition|(Obtaining marginal pdf from joint pdf)
For continuous random variables <math>X_1,\dotsc,X_n</math> with joint pdf <math>f</math>, the marginal pdf of <math>X_i</math> is
<math display=block>
f_{X_i}({\color{red}x})=\underbrace{\int_{-\infty}^{\infty}\cdots\int_{-\infty}^{\infty}}_{n-1\;\text{integrations}}f(u_1,\dotsc,u_{i-1},{\color{red}x},u_{i+1},\dotsc,u_n)\,du_1\cdots\,du_{i-1}\,du_{i+1}\cdots\,du_n.
</math>}}
{{colored proof|
Recall the proposition about obtaining marginal cdf from joint cdf. We have
<math display=block>
\begin{align}
&& F_{X_i}({\color{red}x})&=F(\infty,\dotsc,\infty,\overbrace{\color{red}x}^{i\text{-th position}},\infty,\dotsc,\infty)\\
&\Rightarrow& \int_{-\infty}^{\color{red}x} f_{X_i}(u)\,du&=\int_{-\infty}^{\infty}\cdots
\int_{-\infty}^{\color{red}x}\cdots\int_{-\infty}^{\infty}f(u_1,\dotsc,u_n)\,du_n\cdots\,du_i\cdots\,du_1\qquad\text{by definitions}\\
&\Rightarrow& \frac{d}{dx}\int_{-\infty}^{\color{red}x} f_{X_i}(u)\,du
&=\frac{d}{dx}\int_{-\infty}^{\infty}\cdots\int_{-\infty}^{\color{red}x}\cdots\int_{-\infty}^{\infty}f(u_1,\dotsc,u_n)\,du_n\cdots\,du_i\cdots\,du_1\\
&\Rightarrow& f_{X_i}({\color{red}x})&=\underbrace{\int_{-\infty}^{\infty}\cdots\int_{-\infty}^{\infty}}_{n-1\;\text{integrations}}
f(u_1,\dotsc,u_{i-1},{\color{red}x},u_{i+1},\dotsc,u_n)\,du_1\cdots\,du_{i-1}\,du_{i+1}\cdots\,du_n\qquad\text{by fundamental theorem of calculus}
\end{align}
</math>
}}
{{colored proposition|(Obtaining joint pdf from joint cdf)
If a joint cdf <math>F</math> of jointly continuous random variables has each {{colored em|partial deriviative}} at <math>(x_1,\dotsc,x_n)</math>, then the joint pdf is
<math display=block>
f(x_1,\dotsc,x_n)=\frac{\partial^n}{\partial x_1\cdots\partial x_n}F(x_1,\cdots,x_n).
</math>}}
{{colored proof|
It follows from using fundamental theorem of calculus <math>n</math> times.
}}
{{colored example|
If the joint pdf of jointly continuous random variable <math>(X,Y)</math> is
<math display=block>
f(x,y)=\frac{1}{4}xy(\mathbf 1\{x,y\in[0,1]\}),
</math>
the marginal pdf of <math>X</math> is
<math display=block>
f_X(x)=\int_{-\infty}^{\infty}\frac{1}{4}xy(\mathbf 1\{x\in[0,1]\}\mathbf 1\{y\in[0,1]\})\,dy
=\frac{1}{4}x(\mathbf 1\{x\in[0,1]\})\underbrace{\int_0^1 y\,dy}_{1^2/2-0^2/2}
=\frac{1}{8}x(\mathbf 1\{x\in[0,1]\}).
</math>
Also,
<math display=block>
\mathbb P((X,Y)\le(1/2,1/2))=\int_{-\infty}^{1/2}\int_{-\infty}^{1/2}\frac{1}{4}xy(\mathbf 1\{x,y\in[0,1]\}\,dx\,dy
=\frac{1}{4}\int_0^{1/2}y\int_0^{1/2}x\,dx\,dy
=\frac{1}{4}\cdot\frac{(1/2)^2}{2}\int_0^{1/2}y\,dy
=\frac{1}{4}\cdot\frac{1}{8}\cdot\frac{1}{8}
=\frac{1}{256}.
</math>
}}
{{colored exercise|
Let <math>X</math> and <math>Y</math> be jointly continuous random variables.
Consider the joint cdf of <math>(X,Y)</math>:
<math display=block>
F(x,y)=\mathbf 1\{x,y\in[0,2]\}\frac{x^2y^3}{32}.
</math>
<quiz display=simple>
{Calculate the joint pdf of <math>(X,Y)</math>. 
|type="()"}
- <math>f(x,y)=\mathbf 1\{x,y\in[0,2]\}\frac{3xy^2}{8}</math> 
+ <math>f(x,y)=\mathbf 1\{x,y\in[0,2]\}\frac{3xy^2}{16}</math> 
||<math>f(x,y)=\frac{\partial^2}{\partial x\partial y}\left(\mathbf 1\{x,y\in[0,2]\}\frac{x^2y^3}{32}\right)=\frac{\partial}{\partial x}\left(\mathbf 1\{x,y\in[0,2]\}\frac{3x^2y^2}{32}\right)=\mathbf 1\{x,y\in[0,2]\}\frac{3xy^2}{16}.</math>
- <math>f(x,y)=\mathbf 1\{x,y\in[0,2]\}\frac{xy^2}{16}</math> 
- <math>f(x,y)=\mathbf 1\{x,y\in[0,2]\}\frac{xy^2}{8}</math> 

{Calculate the marginal pdf of <math>X</math>.
|type="()"}
- <math>\mathbf 1\{0\le y\le 2\}\frac{3}{8}y^2</math> 
|| We are not computing marginal pdf of <math>Y</math>. 
+ <math>\mathbf 1\{0\le x\le 2\}\frac{1}{2}x</math> 
|| Marginal pdf of <math>X</math> is <math>\int_{-\infty}^{\infty}\mathbf 1\{x,y\in[0,2]\}\frac{3xy^2}{16}\,dy=\mathbf 1\{0\le x\le 2\}(3x/16)\underbrace{\int_0^2 y^2\,dy}_{8/3}=\mathbf 1\{0\le x\le 2\}\frac{1}{2}x.</math> 
- <math>\mathbf 1\{0\le y\le 2\}\frac{1}{12}y^3</math> 
|| We are not computing marginal cdf of <math>Y</math>. 
- <math>\mathbf 1\{0\le x\le 2\}\frac{1}{8}x^2</math> 
|| We are not computing marginal cdf of <math>X</math>. 
</quiz>
}}

==Independence==
Recall that multiple events are independent if the probability for the intersection of them equals the product of probabilities of each event, by definition.
Since <math>\{X\in A\}</math> is also an event, we have a natural definition of independence for {{colored em|random variables}} as follows:
{{colored definition|
(Independence of random variables)
Random variables <math>X_1,X_2,\dotsc,X_n</math> are {{colored em|independent}} if 
<math display=block>
\mathbb P(X_1\in A_1\cap\cdots\cap X_n\in A_n)=\mathbb P(X_1\in A_1)\cdots\mathbb P(X_n\in A_n)
</math>
for each <math>n</math> and for each subset <math>A_1,A_2,\dotsc,A_n\subseteq\mathbb R</math>.
}}
{{colored remark|
Under this condition, the {{colored em|events}} <math>\{X_1\in A_1\},\dotsc,\{X_n\in A_n\}</math> are independent.
}}
{{colored theorem|
(Alternative condition for independence of random variables)
Random variables <math>X_1,X_2,\dotsc,X_n</math> are {{colored em|independent}} {{colored em|if and only if}}
the joint cdf of <math>(X_1,\dotsc,X_n)</math> 
<math display=block>
F(x_1,\dotsc,x_n)=F_{X_1}(x_1)\cdots F_{X_n}(x_n)
</math>
or
the joint pdf or pmf of <math>(X_1,\dotsc,X_n)</math> 
<math display=block>
f(x_1,\dotsc,x_n)=f_{X_1}(x_1)\cdots f_{X_n}(x_n)
</math>
for each <math>x_1,\dotsc,x_n\in\mathbb R</math>. 
}}
{{colored proof|
Partial:

Only if part:
If random variables <math>X_1,X_2,\dotsc,X_n</math> are independent,
<math display=block>
\mathbb P(X_1\in A_1\cap\cdots\cap X_n\in A_n)=\mathbb P(X_1\in A_1)\cdots\mathbb P(X_n\in A_n)
</math>
for each <math>n</math> and for each subset <math>A_1,A_2,\dotsc,A_n\subseteq\mathbb R</math>. 
Setting <math>A_1=(-\infty,x_1),\dotsc,A_n=(-\infty,x_n)</math>, and we have
<math display=block>
\mathbb P(X_1\le x_1\cap\cdots\cap X_n\le x_n)=\mathbb P(X_1\le x_1)\cdots\mathbb P(X_n\le x_n)
\implies
F(x_1,\dotsc,x_n)=F_{X_1}(x_1)\cdots F_{X_n}(x_n).
</math>
Thus, we obtain the result for the joint cdf part.

For the joint pdf part,
<math display=block>
\begin{align}
&& F(x_1,\dotsc,x_n)&=F_{X_1}(x_1)\cdots F_{X_n}(x_n)\\
&\Rightarrow& \frac{\partial^n}{\partial x_1\cdots\partial x_n}F(x_1,\dotsc,x_n)
&=\frac{\partial^n}{\partial x_1\cdots\partial x_n}
\left(F_{X_1}(x_1)\cdots F_{X_n}(x_n)\right)\\
&\Rightarrow& f(x_1,\dotsc,x_n)&=f_{X_n}(x_n)\frac{\partial^n}{\partial x_1\cdots\partial x_{n-1}}\left(F_{X_1}(x_1)\cdots F_{X_{n-1}}(x_{n-1})\right)\\
&& &=f_{X_n}(x_n)f_{X_{n-1}}(x_{n-1})
\frac{\partial^n}{\partial x_1\cdots\partial x_{n-2}}\left(F_{X_1}(x_1)\cdots F_{X_{n-2}}(x_{n-2})\right)\\
&& &=\cdots=f_{X_1}(x_1)\cdots f_{X_n}(x_n)
\end{align}
</math>
}}
{{colored remark|
* That is, if joint cdf (joint pdf (pmf)) can be factorized as the product of marginal cdf's (marginal pdf's (pmf's))
* Actually, if we can factorize the joint cdf or joint pdf or joint pmf as the product of some functions in each of the variables, then the condition is also satisfied.
}}
{{colored example|
The joint pdf of two {{colored em|independent}} exponential random variables with rate <math>\lambda</math>, <math>X</math> and <math>Y</math> is
<math display=block>
f(x,y)=(\mathbf 1\{x\ge 0\}\lambda e^{-\lambda x})(\mathbf 1\{y\ge 0\}\lambda e^{-\lambda y})
=\mathbf 1\{x,y\ge 0\}\lambda^2e^{-\lambda(x+y)}.
</math>
(Random variables <math>X</math> and <math>Y</math> are said to be {{colored em|independent and identically distributed}} (i.i.d.) in this case) 

In general, the joint pdf of <math>n</math> {{colored em|independent}} exponential random variables with rate <math>\lambda</math>, <math>X_1,\dotsc,X_n</math> is
<math display=block>
f(x_1,\dotsc,x_n)=\mathbf 1\{x_1,\dotsc,x_n\ge 0\}\lambda^ne^{-\lambda(x_1+\cdots+x_n)}.
</math>
(Random variables <math>X_1,\dotsc,X_n</math> are also i.i.d. in this case) 

On the other hand, if the joint pdf of two random variables <math>V</math> and <math>W</math> are
<math display=block>
f(v,w)=\mathbf 1\{w\le 2-2v\},
</math>
random variables <math>V</math> and <math>W</math> are {{colored em|dependent}} since the joint pdf cannot be factorized as the product of marginal pdf's.
}}
{{colored exercise|
Let <math>X,Y,Z</math> be jointly continuous random variables.
Consider a joint pdf of <math>(X,Y,Z)</math>: 
<math display=block>
f(x,y,z)=\mathbf 1\{x,y,z\ge 0\}\mathbf 1\{x+y+z/k\le 1\}.
</math>
<quiz display=simple>
{Calculate <math>k</math>. 
|type="()"}
- 1
+ 2
- 3
- 4

{Are <math>X,Y,Z</math> independent?
|type="()"}
- yes
+ no
|| since we cannot factorize the joint pdf as the product of the marginal pdf's.
</quiz>

Consider another joint pdf of <math>(X,Y,Z)</math>:
<math display=block>
f(x,y,z)=\mathbf 1\{x,y,z\ge 0\}\mathbf 1\{y\le 1-x\}\mathbf 1\{z\le k\}
</math>

<quiz display=simple>
{Calculate <math>k</math>. 
|type="()"}
- 1
+ 2
- 3
- 4

{Are <math>X,Y,Z</math> independent?
|type="()"}
- yes
+ no
|| since we cannot factorize the joint pdf as the product of the marginal pdf's.
</quiz>

Consider another joint pdf of <math>(X,Y,Z)</math>: 
<math display=block>
f(x,y,z)=kxyz\mathbf 1\{x,y\in[0,1]\}\mathbf 1\{z\in[0,2]\}.
</math>
<quiz display=simple>
{Calculate <math>k</math>. 
|type="()"}
- 1
+ 2
- 3
- 4

{Are <math>X,Y,Z</math> independent?
|type="()"}
+ yes
|| We can factorize the joint pdf as the product of three functions in variable <math>x,y,z</math> respectively: <math>f(x,y,z)=(2x\mathbf 1\{x\in[0,1]\})(y\mathbf 1\{y\in[0,1]\})(z\mathbf 1\{z\in[0,2]\})</math> 
|| Indeed, <math>f_X(x)=2x\mathbf 1\{x\in[0,1]\}\int_0^1 y\int_0^2 z\,dz\,dy=2x\mathbf 1\{x\in[0,1]\}</math>, <math>f_Y(y)=2y\mathbf 1\{y\in[0,1]\}\int_0^1 x\int_0^2 z\,dz\,dx=2y\mathbf 1\{y\in[0,1]\}</math> and <math>f_Z(z)=2z\mathbf 1\{z\in[0,2]\}\int_0^1 x\int_0^1 y\,dy\,dx=z/2\mathbf 1\{z\in[0,2]\}</math> 
|| So, <math>f(x,y,z)=(2x\mathbf 1\{x\in[0,1]\})(2y\mathbf 1\{y\in[0,1]\})(z/2\mathbf 1\{z\in[0,2]\})=f_X(x)f_Y(y)f_Z(z)</math> 
- no
</quiz>
}}
{{colored proposition|
(Independence of events concerning disjoint sets of independent random variables)
Suppose random variables <math>X_1,X_2,\dotsc</math> are independent. Then, 
for each <math>r<s<t<\cdots</math> and fixed functions <math>f_1,f_2,f_3,\dotsc</math>,
the random variables 
<math display=block>
Y_1=f_1(X_1,\dotsc,X_{\color{red}r}),\quad
Y_2=f_2(X_{{\color{red}r}+1},\dotsc,X_{\color{blue}s}),\quad
Y_3=f_3(X_{{\color{blue}s}+1},\dotsc,X_t),\dotsc
</math>
are independent.
}}
{{colored example|
Suppose <math>X_1,X_2,X_3,X_4</math> are independent Bernoulli random variables with success probability <math>p</math>.
Then, 
<math>Y_1=X_1+X_2</math> and <math>Y_2=X_3-X_4</math> are also independent.

On the other hand,
<math>Y_1=X_1+X_2</math> and <math>Y_2=2-X_3-X_2</math> are {{colored em|not}} independent. A counter-example to the independence is
<math display=block>
\underbrace{\mathbb P(Y_1=2\cap Y_2=2)}_0\ne\underbrace{\mathbb P(Y_1=2)\mathbb P(Y_2=2)}_{\text{may}\;\ne 0}.   
</math>
Left hand side equals zero since <math>Y_1=2\implies X_2=1</math>, but <math>Y_2=2\implies X_2=0</math>.  

Right hand side may not equal zero since
<math>\mathbb P(Y_1=2)=\mathbb P(X_1=1\cap X_2=1)=p^2</math>, 
and <math>\mathbb P(Y_2=2)=\mathbb P(X_2=0\cap X_3=0)=(1-p)^2</math>.  
We can see that <math>p^2(1-p^2)</math> may not equal zero.
}}
{{colored exercise|
<quiz display=simple>
{Let <math>X_1,\dotsc,X_n</math> be i.i.d. random variables, and <math>Y_1,\dotsc,Y_n</math> also be i.i.d. random variables. Which of the following is (are) true?
|type="[]"}
+ <math>\sum_{i=1}^{n-1}X_i</math> and <math>X_n</math> are independent.
+ <math>X_1^{X_2}</math> and <math>X_3^{X_4}</math> are independent.
- <math>\prod_{i=1}^n X_i</math> and <math>\prod_{i=1}^n Y_i</math> are independent.
|| <math>X_1,\dotsc,X_n,Y_1,\dotsc,Y_n</math> may not be independent. It is not given.
+ <math>X_1+X_2+X_3</math> and <math>Y_1+Y_2+Y_3</math> are independent if <math>X_1,\dotsc,X_n,Y_1</math> are independent.
|| Since <math>Y_1,\dotsc,Y_n</math> are i.i.d., 
|| if <math>X_1,\dotsc,X_n,Y_1</math> are independent, then 
|| <math>X_1,\dotsc,X_n,Y_1,\dotsc,Y_n</math> are independent.
</quiz>
}}
===Sum of independent random variables (optional)===
In general, we use joint cdf, pdf or pmf to determine the distribution of sum of independent random variables by first principle. 
In particular, there are some interesting results related to the distribution of {{colored em|sum}} of {{colored em|independent}} random variables.
{{Collapse top|Sum of independent random variables}}
{{colored proposition|
(Convolution of cdf's and pdf's)
If the cdf of independent random variables <math>X</math> and <math>Y</math> are <math>F_X</math> and <math>F_Y</math> respectively, 
then the cdf of <math>X+Y</math> is
<math display=block>{\color{red}F}_{X+Y}(z)=\int_{-\infty}^{\infty}{\color{red}F}_X(z-y)f_Y(y)\,dy,</math>
and the pdf of <math>X+Y</math> is
<math display=block>{\color{blue}f}_{X+Y}(z)=\int_{-\infty}^{\infty}{\color{blue}f}_X(z-y)f_Y(y)\,dy.</math>
}}
{{colored proof|
* Continuous case:
:* cdf: <math display=block>
\begin{align}
F_{X+Y}(z)&=\mathbb P(X+Y\le z)&\text{by definition}\\
&=\iint_{x+y\le z}f_X(x)f_Y(y)\,dx\,dy&\text{by definition and independence}\\
&=\int_{-\infty}^{\infty}\int_{-\infty}^{z-y}f_X(x)f_Y(y)\,dx\,dy&\text{by Fubini's theorem}\\
&=\int_{-\infty}^{\infty}\left(\int_{-\infty}^{z-y}f_X(x)\,dx\right)f_Y(y)\,dy\\
&=\int_{-\infty}^{\infty}F_X(z-y)f_Y(y)\,dy&\text{by definition}.
\end{align}
</math>
<pre>
/\                                     
//\ y                                
///\|
////*
////|\
////|/\
////|//\ x+y=z <=> x=z-y
////|///\
////|////\
----*-----*--------------- x 
////|//////\
////|///////\

-->: -infty to z-y
^
|: -infty to infty
 
*--*
|//| : x+y <= z
*--*
</pre>
:* pdf: <math display=block>
\begin{align}
f_{X+Y}(z)&=\frac{d}{dz}\int_{-\infty}^{\infty}F_X(z-y)f_Y(y)\,dy\\
&=\int_{-\infty}^{\infty}\frac{d}{dz}F_X(z-y)f_Y(y)\,dy&\text{by fundamental theorem of calculus}\\
&=\int_{-\infty}^{\infty}f_X(z-y)f_Y(y)\,dy.
\end{align}
</math>
}}
{{colored remark|
* The cdf and pdf in this case are actually the {{colored em|convolution}} of the cdf's <math>F_X</math> and <math>F_Y</math>, and pdf's (pmf's) <math>f_X</math> and <math>f_Y</math> respectively, and hence the name of the proposition. 
}}
{{colored example|
* Let the pdf of <math>X</math> be <math>f(x)=\mathbf 1\{0\le {\color{blue}x}\le 1\}</math>. 
* Let the pdf of <math>Y</math> be <math>f(y)=\mathbf 1\{-1\le y\le 0\}</math>. 
* Then, the pdf of <math>X+Y</math> is
<math display=block>
\begin{align}
f_{X+Y}(z)&=\int_{-\infty}^{\infty}\mathbf 1\{0\le {\color{blue}z-y}\le 1\}\mathbf 1\{-1\le y\le 0\}\,dy\\
&=\int_{-\infty}^{\infty}\mathbf 1\{z-1\le y\le z\}\mathbf 1\{-1\le y\le 0\}\,dy\\
&=\mathbf 1\{0\le z\le 1\}\int_{z-1}^{0}\,dy+\mathbf 1\{-1\le z\le 0\}\int_{z}^{0}\,dy\\
&=\mathbf 1\{0\le z\le 1\}(1-z)-z\mathbf 1\{-1\le z\le 0\}.
\end{align}
</math>
Graphically, the pdf looks like
<pre>
        y
        |
        |
        |
     *  * 1
      \ |\  
  y=-z \| \ y=1-z
-----*--*--*----- z
    -1 O|  1   
        |
     -1 *
        |
</pre>
{{colored exercise|
<quiz display=simple>
{Calculate <math>\mathbb P(X=2-Y)</math>. 
|type="()"}
+ 0
|| <math>\mathbb P(X=2-Y)=\mathbb P(X+Y=2)=0</math> since <math>X</math> and <math>Y</math> are continuous.
- 1/4
- 1/2
- 3/4
- 1

{Calculate <math>\mathbb P(X<2-Y)</math>. 
|type="()"}
- 0
- 1/4
- 1/2
- 3/4
+ 1
|| <math>\mathbb P(X<2-Y)=\mathbb P(X+Y<2)=\mathbb P(X+Y\in(-\infty,2))</math>.
|| Since <math>(-\infty,2)</math> covers the whole support of <math>f_{X+Y}</math> (which is <math>[-1,1]</math>, by observing the indicator functions), the probability is one.  

{Calculate <math>k</math> such that <math>\mathbb P(X<k-Y)=1/2</math>. 
|type="()"}
- -1/2
- -1/4
+ 0
|| <math>\mathbb P(X<k-Y)=\mathbb P(X+Y<k)=\int_{-\infty}^k\mathbf 1\{0\le z\le 1\}(1-z)-z\mathbf 1\{-1\le z\le 0\}\,dz</math>. 
|| We may notice that <math>\int_{-1}^0 -z\,da=1/2</math>, and immediately conclude that the answer is 0. 
|| If we do not notice that, we can first {{colored em|assuming}} that <math>k\le 0</math>, so that <math>\mathbf 1\{0\le z\le 1\}=0</math>, and thus the calculation is simplified, then for each <math>k</math>, <math>\mathbb P(X<k-Y)=\int_{-\infty}^k-z\mathbf 1\{-1\le z\le 0\}\,dz=-\int_{-1}^k z\,dz=-k^2/2+(-1)^2/2=1/2-k^2/2.</math>  
|| Setting this to equal <math>1/2</math>, we can solve that <math>k=0</math>. 
- 1/4
- 1/2
</quiz>
}}
}}
{{colored proposition|
(Convolution of pmf's)
If the pmf of independent random variables <math>X</math> and <math>Y</math> are <math>f_X</math> and <math>f_Y</math> respectively, then the pmf of <math>X+Y</math> is
<math display=block>f_{X+Y}(n)=\sum_{k=0}^{n}f_X(k)f_Y(n-k)=f_X(0)f_Y(n)+f_X(1)f_Y(n-1)+\dotsb+f_X(n-1)f_Y(1)+f_X(n)f_Y(0).</math>
}}
{{colored proof|
* Let <math>E_i=\{X=i\}\cap\{Y=n-i\}</math>. 
* For each nonnegative integer <math>n</math>, 
<math display=block>\{X+Y=n\}=E_0\cup E_1\cup\dotsb\cup E_n.</math>
* Since <math>E_i\cap E_j=\varnothing</math> for each <math>i\ne j</math>, <math>E_i</math>'s are pairwise disjoint.
* Hence, by extended P3 and independence of <math>X</math> and <math>Y</math>, <math display=block>\mathbb P(X+Y=n)=\mathbb P(X=0)\mathbb P(Y=n)+\mathbb P(X=1)\mathbb P(Y=n-1)+\dotsb+\mathbb P(X=n)\mathbb P(Y=0).</math>
* The result follows by definition.
}}
{{colored example|
We roll a fair six-faced dice twice (independently).
Then, the probability for the sum of the numbers coming up to be 7 is <math>\underbrace{(1/6)(1/6)+\dotsb+(1/6)(1/6)}_{6\text{ times}}=1/6</math>.
{{colored proof|
Let <math>X</math> and <math>Y</math> be the first and second number coming up respectively.
The desired probability is
<math display=block>\mathbb P(X+Y=7)=\underbrace{\mathbb P(X=1)}_{1/6}\underbrace{\mathbb P(Y=6)}_{1/6}+\dotsb+\underbrace{\mathbb P(X=6)}_{1/6}\underbrace{\mathbb P(Y=1)}_{1/6}=\underbrace{(1/6)(1/6)+\dotsb+(1/6)(1/6)}_{6\text{ times}}=1/6.</math>
}}
{{colored exercise|
<quiz display=simple>
{Calculate the probability for the sum to be 6 instead.
|type="()"}
- 1/12
|| <math>\mathbb P(X=4)\mathbb P(Y=2)</math> and <math>\mathbb P(X=5)\mathbb P(Y=1)</math> also need to be considered.
- 1/6
|| The sum is 6 now, not 7.
+ 5/36
|| <math display=block>\underbrace{\mathbb P(X=1)}_{1/6}\underbrace{\mathbb P(Y=5)}_{1/6}+\dotsb+\underbrace{\mathbb P(X=5)}_{1/6}\underbrace{\mathbb P(Y=1)}_{1/6}=\underbrace{(1/6)(1/6)+\dotsb+(1/6)(1/6)}_{5\text{ times}}=5/36.</math>
- 7/36
- 4/9

{The probability for the sum to be <math>k</math> is 0. Which of the following is (are) possible value(s) of <math>k</math>? 
|type="[]"}
+ 1
|｜The minimum possible sum is 2 (achieved by <math>X=Y=1</math>). 
- 2
- 3
- 12
+ 13
|｜The maximum possible sum is 12 (achieved by <math>X=Y=6</math>). 

{Suppose the dice is loaded such that the probability for the number coming up to be 6 is now <math>1/2</math>, and for other numbers, they are equally likely to be coming up. Calculate the probability for the sum to be 7 now.
|type="()"}
- 0.1
- 0.101
- 0.1001
+ 0.10000001
|| <math>\mathbb P(X+Y=7)=\underbrace{\mathbb P(X=1)}_{1/10}\underbrace{\mathbb P(Y=6)}_{1/2}+\dotsb+\underbrace{\mathbb P(X=6)}_{1/2}\mathbb P(Y=1)}_{1/10}=\underbrace{(1/10)(1/2)+\big((1/10)(1/10)\big)^4+(1/2)(1/10)=0.10000001.</math> 
- 0.167
|| The probabilities for other numbers coming up change.
</quiz>
}}
}}
{{colored proposition|
(Sum of independent Poisson r.v.'s)
If <math>X_1\sim\operatorname{Pois}(\lambda_1),\dotsc,X_n\sim\operatorname{Pois}(\lambda_n)</math> and <math>X_1,\dotsc,X_n</math> are independent, then <math>X_1+\dotsb+X_n\sim\operatorname{Pois}(\lambda_1+\dotsb+\lambda_n)</math>. 
}}
{{colored proof|
* The pmf of <math>X_1+X_2</math> is
<math display=block>
\begin{align}
f_{X_1+X_2}(a)&=\sum_{k=0}^{n}\frac{e^{-\lambda_1}\lambda_1^{k}}{k!}\cdot\frac{e^{-\lambda_2}\lambda_2^{n-k}}{(n-k)!}&\text{by the proposition about convolution of pmf's}\\
&=e^{-\lambda_1-\lambda_2}\sum_{k=0}^{n}\frac{\lambda_1^k\cdot \lambda_2^{n-k}}{k!(n-k)!}\\
&=\frac{e^{-(\lambda_1+\lambda_2)}}{n!}\underbrace{\sum_{k=0}^{n}\frac{n!}{k!(n-k)!}\cdot \lambda_1^k\cdot \lambda_2^{n-k}}_{=(\lambda_1+\lambda_2)^n}&\text{ by binomial theorem}.\\
\end{align}
</math>
* This pmf as the pmf of <math>\operatorname{Pois}(\lambda_1+\lambda_2)</math>, and so <math>X_1+X_2\sim\operatorname{Pois}(\lambda_1+\lambda_2)</math>. 
* We can extend this result to <math>n</math> Poisson r.v.'s by induction.
}}
{{colored example|
There are two service counters, for which the first one receives <math>X\sim\operatorname{Pois}(3)</math> enquiries per hour, while the second one receives <math>Y\sim\operatorname{Pois}(4)</math> enquiries per hour. 
Given that <math>X</math> and <math>Y</math> are independent, 
the number of enquiries received by the two counters per hour follows <math>\operatorname{Pois}(3+4)=\operatorname{Pois}(7)</math>. 
{{colored proof|
* The number of enquiries received by the two counters per hour is <math>X+Y</math>. 
* Then, the result follows from the proposition about sum of Poisson r.v.'s.
}}
{{colored exercise|
<quiz display=simple>
{Which distribution does the number of enquiries received by the first counter for two hours follow?
|type="()"}
- <math>\operatorname{Pois}(3)</math> 
- <math>\operatorname{Pois}(4)</math> 
+ <math>\operatorname{Pois}(6)</math> 
|| The number is given by <math>2X=X+X</math>, so the number follows <math>\operatorname{Pois}(3+3)=\operatorname{Pois}(6)</math>.
- <math>\operatorname{Pois}(7)</math> 
- <math>\operatorname{Pois}(8)</math> 
</quiz>
}}
}}
{{Collapse bottom}}

===Order statistics===
{{colored definition|
(Order statistics)
Let <math>X_1,\dotsc,X_n</math> be <math>n</math> i.i.d. r.v.'s (each with cdf <math>F(x)</math>). 
Define <math>X_{(1)},X_{(2)},\dotsc,X_{(n)}</math> be the smallest, second smallest, ..., largest of <math>X_1,X_2,\dotsc,X_n</math>. 
Then, the ordered values <math>X_{(1)}\le X_{(2)}\le\dotsb\le X_{(n)}</math> is the {{colored em|order statistics}}. 
}}
{{colored proposition|
(Cdf of order statistics)
The cdf of <math>X_{(k)}</math> (<math>k</math> is an integer such that <math>1\le k\le n</math>) is 
<math display=block>F_{X_{(k)}}({\color{blue}x})=\sum_{j=k}^{n}\binom{n}{j}(F({\color{blue}x}))^j\big(1-F({\color{blue}x})\big)^{n-j}.</math>
}}
{{colored proof|
* Consider the event <math>\{X_{(k)}\le{\color{blue}x}\}</math>.
<pre>
                          Possible positions of x
                      |<--------------------->
    *---*----...------*----*------...--------*
X  (1)  (2)          (k)  (k+1)             (n)
                      |----------------------> when x moves RHS like this, >=k X_i are at the LHS of x
</pre>
* We can see from the above figure that <math>\{X_{(k)}\le {\color{blue}x}\}=\{\text{at least }k\text{ of the }X_i\text{'s are }\le{\color{blue}x}\}</math>. 
* Let no. of <math>X_i</math>'s that are less than or equal to <math>{\color{blue}x}</math> be <math>N</math>. 
* Since <math>N\sim\operatorname{Binom}(n,\mathbb P(X_i\le{\color{blue}x}))\overset{\text{ def }}=\operatorname{Binom}(n,F({\color{blue}x}))</math> (because for each <math>X_i</math>, we can treat <math>X_i\le x</math> and <math>X_i>x</math> be the two outcomes in a Bernoulli trial), 
* The cdf is
<math display=block>\mathbb P(X_{(k)}\le {\color{blue}x})=\mathbb P(N\ge k)=\sum_{j=k}^{n}\binom{n}{j}(F({\color{blue}x}))^j\big(1-F({\color{blue}x})\big).</math>
}}
{{colored example|
Let <math>X_1,X_2,X_3</math> be i.i.d. r.v.'s following <math>\operatorname{Exp}(2)</math>. 
Then, the cdf of <math>X_{(2)}</math> is
<math display=block>\sum_{j=2}^{3}\binom{3}{j}(F(x))^j(1-F(x))^{3-j}=\mathbf 1\{x\ge 0\}\left(\binom{3}{2}(1-e^{-2x})^2(e^{-2x})+\binom{3}{3}(1-e^{-2x})^3\right)=\mathbf 1\{x\ge 0\}\left(3(1-e^{-2x})^2(e^{2x})+(1-e^{-2x})^3\right).</math>
{{colored exercise|
<quiz display=simple>
{Calculate <math>\mathbb P(X_{(2)}\ge 2)</math>. 
|type="()"}
- 0.000665
+ 0.000994
|| <math>\mathbb P(X_{(2)}\ge 2)=\mathbb P(X_{(2)}>2)=1-\mathbb P(X_{(2)}\le 2)=1-F_{X_{(2)}}(2)=1-(3(1-e^{-4})^2(e^{-4}+1-e^{-4})\approx 0.000994.</math> 
|| (Recall that <math>X_{(2)}</math> is a continuous r.v., so the probability for it to equal a specific value is zero.)
- 0.036296
- 0.963704
- 0.999335
</quiz>
}}
}}

==Poisson process==
{{colored definition|
[[File:Poisson_process.svg|thumbnail|Illustration of Poisson process. Each circle indicates one arrival. The arrivals occur at common rate <math>\lambda</math>, and the successive interarrivial times are independent.]]
If successive {{colored em|interarrival}} times of unpredictable events are {{colored em|independent}} random variables, 
with each following an {{colored em|exponential}} distribution with a common rate <math>\lambda</math>, 
then the process of arrivals is a {{colored em|Poisson process}} with rate <math>\lambda</math>. 
}}
There are several important properties for Poisson process.
{{colored proposition|
(Time to <math>n</math>-th event in Poisson process)
The time to <math>n</math>-th event  in a Poisson process follows 
the <math>\operatorname{Gamma}(n,\lambda)</math> distribution.
}}
{{colored proof|
* The time to <math>n</math>-th event is <math>X_1+\dotsb+X_n</math>, with each following <math>\operatorname{Exp}(\lambda)</math>.
* It suffices to prove that <math>X_1+X_2\sim\operatorname{Gamma}(2,\lambda)</math>, and then the desired result follows by induction.
* <math display=block>
\begin{align}
f_{X_1+X_2}(z)&=\lambda^2\int_{-\infty}^{\infty}\mathbf 1\{\underbrace{z-x\ge 0}_{x\le z}\}\mathbf 1\{x\ge 0\}e^{-\lambda(z-x)}e^{-\lambda x}\,dx&\text{by proposition about convolution of pdf's}\\
&=\lambda^2\int_{0}^{z}e^{-\lambda(z\cancel{-x})\cancel{-\lambda x}}\,dx\\
&=\lambda^2\int_{0}^{z}e^{-\lambda z}\,dx\\
&=\lambda^2ze^{-\lambda z}\\
&=\frac{\lambda^2ze^{-\lambda z}}{\Gamma(2)}&\text{since }\Gamma(2)=1!=1,
\end{align}
</math>
: which is the pdf of <math>\Gamma(2,\lambda)</math>, as desired.
}}
{{colored remark|
The time to <math>n</math>-th event is also the sum of the <math>n</math> successive interarrival times before the <math>n</math>-th event. 
}}
{{colored proposition|
(Number of arrivals within a fixed time interval)
The number of arrivals within a fixed time interval of length <math>t</math> follows the <math>\operatorname{Pois}(\lambda t)</math> distribution.
}}
{{colored proof|
For each nonnegative integer <math>n</math>, 
let <math>V</math> be the interarrival time between the <math>n</math>-th and <math>n+1</math>-th arrival, and <math>W</math> be the time to <math>n</math>th event, starting from the beginning of the fixed time interval (we can treat the start to be time zero because of the memoryless property).  
The joint pdf of <math>(V,W)</math> is
<math display=block>
\begin{align}
f(v,w)&=f_V(v)f_W(w)&\text{by independence}\\
&=\underbrace{(\lambda e^{-\lambda v})}_{\text{pdf of}\;\operatorname{Exp}(\lambda)}\underbrace{\left(\frac{\lambda^nw^{n-1}e^{-\lambda w}}{(n-1)!}\right)}_{\text{pdf of}\operatorname{Gamma}(n,\lambda)}.
\end{align}
</math>
Let <math>N</math> the number of arrivals within the fixed time interval.
The pmf of <math>N</math> is
<math display=block>
\begin{align}
\mathbb P(N=n)&=\mathbb P(W\le t\cap \underbrace{V+W>t}_{V>t-W})\\
&=\int_0^t\int_{t-w}^{\infty}\underbrace{f(v,w)}_{\text{joint pdf of}\;(V,W)}\,dv\,dw\\
&=\int_0^t\int_{t-w}^{\infty}(\lambda e^{-\lambda v})\left(\frac{\lambda^nw^{n-1}e^{-\lambda w}}{(n-1)!}\right)\,dv\,dw\\
&=\int_0^t\frac{\lambda^nw^{n-1}e^{-\lambda w}}{(n-1)!}\int_{t-w}^{\infty}\lambda e^{-\lambda v}\,dv\,dw\\
&=\frac{\lambda^n}{(n-1)!}\int_0^t w^{n-1}\cancel{e^{-\lambda w}}(0-(-e^{-\lambda(t\cancel{-w})}))\,dw\\
&=\frac{\lambda^n{\color{green}e^{-\lambda t}}}{(n-1)!}\int_0^t w^{n-1}\,dw\\
&=\frac{\lambda^n e^{-\lambda t}}{(n-1)!}\cdot\left(\frac{t^n}{n}-0\right)\\
&=\frac{e^{-\lambda t}(\lambda t)^n}{n!}
\end{align}
</math>
which is the pmf of <math>\operatorname{Pois}(\lambda t)</math>. The result follows. 
}}
{{colored proposition|
(Time to the first arrival with <math>n</math> independent Poisson processes)
Let <math>T_1,T_2,\dotsc,T_n</math> be independent random variables with <math>T_i\sim\operatorname{Exp}(\lambda_i)</math>, in which <math>i=1,2,\dotsc,n</math>. If we define 
<math>T=\min\{T_1,\dotsc,T_n\}</math> (which is the time to the first arrival with <math>n</math> independent Poisson processes), then <math>T\sim\operatorname{Exp}(\lambda_1+\lambda_2+\cdots+\lambda_n)</math>.  
}}
{{colored proof|
For each <math>t>0</math>,
<math display=block>
\begin{align}
&& \mathbb P(T>t)&=\mathbb P(T_1>t\cap\cdots\cap T_n>t)\\
&& &=\mathbb P(T_1>t)\cdots\mathbb P(T_n>t)&\text{by independence}\\
&& &=[1-(\underbrace{1-e^{-\lambda_1 t}}_{\text{cdf of}\;\operatorname{Exp}(\lambda_1)})]
\cdots[1-(\underbrace{1-e^{-\lambda_n t}}_{\text{cdf of}\;\operatorname{Exp}(\lambda_n)})]\\
&& &=e^{-t(\lambda_1+\cdots+\lambda_n)}\\
&\Rightarrow& \mathbb P(T\le t)&=1-e^{-t(\lambda_1+\cdots+\lambda_n)}\\
&\Rightarrow& T&\sim\operatorname{Exp}(\lambda_1+\lambda_2+\cdots+\lambda_n)
\end{align}
</math>
}}
{{colored example|
Suppose there are two service counters, counter A and B, with independent service times following the exponential distribution with rate <math>\lambda</math>.
In the past 10 minutes, John and Peter are being served at counter A and B respectively.

First, the time you need to wait to be served (i.e. the time for one of John and Peter leaves the counter) is the minimum value of the service time for John and Peter counting from now, which are independent and follow the exponential distribution with rate <math>\lambda</math>. 
Thus, your {{colored em|waiting time}} follows the {{colored em|exponential distribution with rate}} <math>\lambda+\lambda=2\lambda</math>. 

Suppose now John leaves the counter A, and you are currently being served at counter A. Then, the {{colored em|probability that you leave the counter first}} is <math>1/2</math>, by {{colored em|memoryless}} property and {{colored em|symmetry}} (the chances that Peter and you leave the counter first are governed by the same chance mechanism), counterintuitively.
}}
{{colored exercise|
Suppose the process of arrivals of car accidents is a Poisson process with unit rate. Let <math>T_i</math> be the time to the <math>i</math>-th  car accidents,
and <math>X_i</math> be the interarrival time between the <math>i-1</math>-th and <math>i</math>-th accidents.
<quiz display=simple>
{ Which of the following is (are) true?
|type="[]"}
+ <math>T_3\sim\operatorname{Gamma}(3,1)</math> 
|| It follows from the proposition about time to <math>n</math>-th event in Poisson process. 
- <math>T_3\sim \operatorname{Exp}(1)</math> 
|| <math>X_i\sim\operatorname{Exp}(1)</math>, not <math>T_i</math>.  
- <math>T_3\sim \operatorname{Exp}(3)</math> 
- <math>T_3\sim \operatorname{Pois}(1)</math> 
|| Poisson distribution concerns about the number of accident, instead of the time.
- <math>T_3\sim \operatorname{Pois}(3)</math> 
|| Poisson distribution concerns about the number of accident, instead of the time.

{Which of the following is (are) true?
|type="[]"}
- <math>X_i\sim\operatorname{Exp}(i)</math> 
|| The Poisson process is with unit rate.
+ <math>X_i\sim\operatorname{Exp}(1)</math> 
|| This follows from the definition of Poisson process.
- <math>X_i\sim\operatorname{Pois}(1)</math> 
|| Poisson distribution concerns about the number of accident, instead of the time.
- <math>X_i-X_{i-1}\sim\operatorname{Exp}(1)</math> 
|| <math>X_{i-1},X_i\sim\operatorname{Exp}(1).</math> But, <math>X_{i}-X_{i-1}</math> do not follow any specific distribution.

{Which of the following is (are) true?
|type="[]"}
+ <math>T_i-T_{i-1}\sim\operatorname{Exp}(1)</math> 
|| <math>T_i-T_{i-1}=X_i\sim\operatorname{Exp}(1)</math> 
+ <math>T_i-T_{i-1}\sim\operatorname{Gamma}(1,1)</math> 
|| <math>T_i-T_{i-1}=X_i\sim\operatorname{Exp}(1)=\operatorname{Gamma}(1,1)</math> by the relationship between exponential and gamma distribution. 
- <math>T_i-T_{i-1}\sim\operatorname{Pois}(1)</math> 
|| Poisson distribution concerns about the number of accident, instead of the time.
- The pmf of the number of arrivals within a fixed time interval of length <math>t</math> is <math>f(x)=\frac{e^{-t}t^x}{x!}</math>. 
|| Since such number of arrivals follows the Poisson distribution with rate <math>1(t)=t</math>,
|| the desired pmf is <math>\frac{e^{-t}t^x}{x!}\mathbf 1\{x\in\{0,1,2,\cdots\}\}</math>. 
</quiz>
}}
<noinclude>
{{nav}}
</noinclude>