Probability/Joint Distributions and Independence

Motivation

Suppose we are given a pmf of a discrete random variable $X$ and a pmf of a discrete random variable $Y$. For example, $$f_X(x)=(\mathrm{𝟏}\{x=0\}+\mathrm{𝟏}\{x=1\})/2\text{and}{f}_Y(y)=(\mathrm{𝟏}\{y=0\}+\mathrm{𝟏}\{y=2\})/2$$ We cannot tell the relationship between $X$ and $Y$ with only such information. They may be related or not related.

For example, the random variable $X$ may be defined as $X=1$ if head comes up and $X=0$ otherwise from tossing a fair coin, and the random variable $Y$ may be defined as $Y=2$ if head comes up and $Y=0$ otherwise from tossing the coin another time. In this case, $X$ and $Y$ are unrelated.

Another possibility is that the random variable $Y$ is defined as $Y=2X$ if head comes up in the first coin tossing, and $Y=0$ otherwise. In this case, $X$ and $Y$ are related.

Yet, in the above two examples, the pmf of $X$ and $Y$ are exactly the same.

Therefore, to tell the Template:Colored em between $X$ and $Y$, we define the Template:Colored em cumulative distribution function, or joint cdf.

Joint distributions

Template:Colored definition Sometimes, we may want to know the random behaviour in one of the random variables involved in a joint cdf. We can do this by computing the marginal cdf from joint cdf. The definition of marginal cdf is as follows: Template:Colored definition Template:Colored remark Template:Colored proposition

Proof. When we set the arguments other than $i$-th argument to be $\mathrm{\infty }$, e.g. $X_1\le \mathrm{\infty }\iff \underset{x\to \mathrm{\infty }}{\lim }{X}_1\le x$ , the joint cdf becomes $$\begin{array}{cc}\mathbb{P}(X_1\le \mathrm{\infty }\cap \cdots \cap X_{i-1}\le \mathrm{\infty }\cap X_i\le x\cap X_{i+1}\le \mathrm{\infty }\cap \cdots \cap X_n\le \mathrm{\infty })& =\underset{1}{\underbrace{\mathbb{P}(X_1\le \mathrm{\infty }\cap \cdots \cap X_{i-1}\le \mathrm{\infty })}}\mathbb{P}(X_i\le x)\underset{1}{\underbrace{\mathbb{P}(X_{i+1}\le \mathrm{\infty }\cap \cdots \cap X_n\le \mathrm{\infty })}}\text{by independence}\\ & =\mathbb{P}(X_i\le x)\\ & =F_{X_i}(x)\end{array}$$ $◻$

Template:Colored remark Template:Colored example Similar to the one-variable case, we have joint pmf and joint pdf. Also, analogously, we have marginal pmf and marginal pdf.

Proof. Consider the case in which there are only two random variables, say $X$ and $Y$. Then, we have $$\sum _{y}f(x,y)=\sum _y\mathbb{P}(X=x\cap Y=y)=\mathbb{P}(X=x)\text{by law of total probability}.$$ Similarly, in general case, we have $$\begin{array}{cc}\sum _{u_n}f(u_1,\dots ,u_{i-1},x,u_{i+1},\dots ,u_n)& =\sum _{u_n}\mathbb{P}(X_1\le u_1\cap \cdots \cap X_{i-1}{u}_{i-1}\cap X_i\le x\cap X_{i+1}\le u_{i+1}\cap \cdots \cap X_{n-1}\le u_{n-1}\cap X_n\le u_n)\\ & =\mathbb{P}(X_1\le u_1\cap \cdots \cap X_{i-1}{u}_{i-1}\cap X_i\le x\cap X_{i+1}\le u_{i+1}\cap \cdots \cap X_{n-1}\le u_{n-1})\text{by law of total probability}.\end{array}$$ Then, we perform similar process on each of the other variables ($n-2$ left), with one extra summation sign added for each process. Thus, in total we will have $n-1$ summation sign, and we will finally get the desired result. $◻$

Template:Colored remark Template:Colored example Template:Colored exercise Template:Colored exercise Template:Hide For Template:Colored em continuous random variables, the definition is generalized version of the one for continuous random variables (univariate case). Template:Colored definition Template:Colored remark Template:Colored definition Template:Colored proposition

Proof. Recall the proposition about obtaining marginal cdf from joint cdf. We have $$\begin{array}{cccc}& & F_{X_i}(x)& =F(\mathrm{\infty },\dots ,\mathrm{\infty },\stackrel{i\text{-th position}}{\overbrace{x}},\mathrm{\infty },\dots ,\mathrm{\infty })\\ & \Rightarrow & \underset{-\mathrm{\infty }}{\overset{x}{\int }}{f}_{X_i}(u)du& =\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\cdots \underset{-\mathrm{\infty }}{\overset{x}{\int }}\cdots \underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}f(u_1,\dots ,u_n)d{u}_n\cdots d{u}_i\cdots d{u}_1\text{by definitions}\\ & \Rightarrow & \frac{d}{dx}\underset{-\mathrm{\infty }}{\overset{x}{\int }}{f}_{X_i}(u)du& =\frac{d}{dx}\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\cdots \underset{-\mathrm{\infty }}{\overset{x}{\int }}\cdots \underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}f(u_1,\dots ,u_n)d{u}_n\cdots d{u}_i\cdots d{u}_1\\ & \Rightarrow & f_{X_i}(x)& =\underset{n-1\text{integrations}}{\underbrace{\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\cdots \underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}}}f(u_1,\dots ,u_{i-1},x,u_{i+1},\dots ,u_n)d{u}_1\cdots d{u}_{i-1}d{u}_{i+1}\cdots d{u}_n\text{by fundamental theorem of calculus}\end{array}$$

$◻$

Proof. It follows from using fundamental theorem of calculus $n$ times.

$◻$

Independence

Recall that multiple events are independent if the probability for the intersection of them equals the product of probabilities of each event, by definition. Since $\{X\in A\}$ is also an event, we have a natural definition of independence for Template:Colored em as follows: Template:Colored definition Template:Colored remark Template:Colored theorem

Proof. Partial:

Only if part: If random variables $X_1,X_2,\dots ,X_n$ are independent, $$\mathbb{P}(X_1\in A_1\cap \cdots \cap X_n\in A_n)=\mathbb{P}(X_1\in A_1)\cdots \mathbb{P}(X_n\in A_n)$$ for each $n$ and for each subset $A_1,A_2,\dots ,A_n\subseteq ℝ$. Setting $A_1=(-\mathrm{\infty },x_1),\dots ,A_n=(-\mathrm{\infty },x_n)$, and we have $$\mathbb{P}(X_1\le x_1\cap \cdots \cap X_n\le x_n)=\mathbb{P}(X_1\le x_1)\cdots \mathbb{P}(X_n\le x_n)⟹F(x_1,\dots ,x_n)=F_{X_1}(x_1)\cdots F_{X_n}(x_n).$$ Thus, we obtain the result for the joint cdf part.

For the joint pdf part, $$\begin{array}{cccc}& & F(x_1,\dots ,x_n)& =F_{X_1}(x_1)\cdots F_{X_n}(x_n)\\ & \Rightarrow & \frac{{\partial }^n}{\partial x_1\cdots \partial x_n}F(x_1,\dots ,x_n)& =\frac{{\partial }^n}{\partial x_1\cdots \partial x_n}(F_{X_1}(x_1)\cdots F_{X_n}(x_n))\\ & \Rightarrow & f(x_1,\dots ,x_n)& =f_{X_n}(x_n)\frac{{\partial }^n}{\partial x_1\cdots \partial x_{n-1}}(F_{X_1}(x_1)\cdots F_{X_{n-1}}(x_{n-1}))\\ & & & =f_{X_n}(x_n)f_{X_{n-1}}(x_{n-1})\frac{{\partial }^n}{\partial x_1\cdots \partial x_{n-2}}(F_{X_1}(x_1)\cdots F_{X_{n-2}}(x_{n-2}))\\ & & & =\cdots =f_{X_1}(x_1)\cdots f_{X_n}(x_n)\end{array}$$

$◻$

Sum of independent random variables (optional)

In general, we use joint cdf, pdf or pmf to determine the distribution of sum of independent random variables by first principle. In particular, there are some interesting results related to the distribution of Template:Colored em of Template:Colored em random variables. Template:Collapse top Template:Colored proposition

Proof.

• Continuous case:

• cdf: $$\begin{array}{ccc}{F}_{X+Y}(z)& =\mathbb{P}(X+Y\le z)& \text{by definition}\\ & =\underset{x+y\le z}{\iint }{f}_X(x)f_Y(y)dxdy& \text{by definition and independence}\\ & =\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\underset{-\mathrm{\infty }}{\overset{z-y}{\int }}{f}_X(x)f_Y(y)dxdy& \text{by Fubini's theorem}\\ & =\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}(\underset{-\mathrm{\infty }}{\overset{z-y}{\int }}{f}_X(x)dx)f_Y(y)dy\\ & =\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}{F}_X(z-y)f_Y(y)dy& \text{by definition}.\end{array}$$

/\                                     
//\ y                                
///\|
////*
////|\
////|/\
////|//\ x+y=z <=> x=z-y
////|///\
////|////\
----*-----*--------------- x 
////|//////\
////|///////\

-->: -infty to z-y
^
|: -infty to infty
 
*--*
|//| : x+y <= z
*--*

• pdf: $$\begin{array}{cc}{f}_{X+Y}(z)& =\frac{d}{dz}\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}{F}_X(z-y)f_Y(y)dy\\ & =\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\frac{d}{dz}{F}_X(z-y)f_Y(y)dy& \text{by fundamental theorem of calculus}\\ & =\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}{f}_X(z-y)f_Y(y)dy.\end{array}$$

$◻$

Proof.

• Let $E_i=\{X=i\}\cap \{Y=n-i\}$.
• For each nonnegative integer $n$,

$$\{X+Y=n\}=E_0\cup E_1\cup \cdots \cup E_n.$$

• Since $E_i\cap E_j=\varnothing$ for each $i\ne j$, $E_i$'s are pairwise disjoint.
• Hence, by extended P3 and independence of $X$ and $Y$, $$\mathbb{P}(X+Y=n)=\mathbb{P}(X=0)\mathbb{P}(Y=n)+\mathbb{P}(X=1)\mathbb{P}(Y=n-1)+\cdots +\mathbb{P}(X=n)\mathbb{P}(Y=0).$$
• The result follows by definition.

$◻$

Proof.

• The pmf of $X_1+X_2$ is

$$\begin{array}{ccc}{f}_{X_1+X_2}(a)& =\sum _{k=0}^n\frac{e^{-{\lambda }_1}{\lambda }_1^k}{k!}\cdot \frac{e^{-{\lambda }_2}{\lambda }_2^{n-k}}{(n-k)!}& \text{by the proposition about convolution of pmf's}\\ & =e^{-{\lambda }_1-{\lambda }_2}\sum _{k=0}^n\frac{{\lambda }_1^k\cdot {\lambda }_2^{n-k}}{k!(n-k)!}\\ & =\frac{e^{-({\lambda }_1+{\lambda }_2)}}{n!}\underset{=({\lambda }_1+{\lambda }_2{)}^n}{\underbrace{\sum _{k=0}^n\frac{n!}{k!(n-k)!}\cdot {\lambda }_1^k\cdot {\lambda }_2^{n-k}}}& \text{ by binomial theorem}.\\ \end{array}$$

• This pmf as the pmf of $\mathrm{Pois}({\lambda }_1+{\lambda }_2)$, and so $X_1+X_2\sim \mathrm{Pois}({\lambda }_1+{\lambda }_2)$.
• We can extend this result to $n$ Poisson r.v.'s by induction.

$◻$

Order statistics

Proof.

• Consider the event $\{X_{(k)}\le x\}$.

                          Possible positions of x
                      |<--------------------->
    *---*----...------*----*------...--------*
X  (1)  (2)          (k)  (k+1)             (n)
                      |----------------------> when x moves RHS like this, >=k X_i are at the LHS of x

• We can see from the above figure that $\{X_{(k)}\le x\}=\{\text{at least }k\text{ of the }{X}_i\text{'s are }\le x\}$.
• Let no. of $X_i$'s that are less than or equal to $x$ be $N$.
• Since $N\sim \mathrm{Binom}(n,\mathbb{P}(X_i\le x))\stackrel{\text{ def }}{=}\mathrm{Binom}(n,F(x))$ (because for each $X_i$, we can treat $X_i\le x$ and $X_i>x$ be the two outcomes in a Bernoulli trial),
• The cdf is

$$\mathbb{P}(X_{(k)}\le x)=\mathbb{P}(N\ge k)=\sum _{j=k}^n(\genfrac{}{}{0ex}{}{n}{j})(F(x){)}^j(1-F(x)).$$

$◻$

Poisson process

Template:Colored definition There are several important properties for Poisson process. Template:Colored proposition

Proof.

• The time to $n$-th event is $X_1+\cdots +X_n$, with each following $\mathrm{Exp}(\lambda )$.
• It suffices to prove that $X_1+X_2\sim \mathrm{Gamma}(2,\lambda )$, and then the desired result follows by induction.
• $$\begin{array}{ccc}{f}_{X_1+X_2}(z)& ={\lambda }^2\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\mathrm{𝟏}\{\underset{x\le z}{\underbrace{z-x\ge 0}}\}\mathrm{𝟏}\{x\ge 0\}{e}^{-\lambda (z-x)}{e}^{-\lambda x}dx& \text{by proposition about convolution of pdf's}\\ & ={\lambda }^2\underset{0}{\overset{z}{\int }}{e}^{-\lambda (z\cancel{-x})\cancel{-\lambda x}}dx\\ & ={\lambda }^2\underset{0}{\overset{z}{\int }}{e}^{-\lambda z}dx\\ & ={\lambda }^{2}z{e}^{-\lambda z}\\ & =\frac{{\lambda }^{2}z{e}^{-\lambda z}}{\mathrm{\Gamma }(2)}& \text{since }\mathrm{\Gamma }(2)=1!=1,\end{array}$$

which is the pdf of $\mathrm{\Gamma }(2,\lambda )$, as desired.

$◻$

Proof. For each nonnegative integer $n$, let $V$ be the interarrival time between the $n$-th and $n+1$-th arrival, and $W$ be the time to $n$th event, starting from the beginning of the fixed time interval (we can treat the start to be time zero because of the memoryless property). The joint pdf of $(V,W)$ is $$\begin{array}{ccc}f(v,w)& =f_V(v)f_W(w)& \text{by independence}\\ & =\underset{\text{pdf of}\mathrm{Exp}(\lambda )}{\underbrace{(\lambda e^{-\lambda v})}}\underset{\text{pdf of}\mathrm{Gamma}(n,\lambda )}{\underbrace{\left(\frac{{\lambda }^n{w}^{n-1}{e}^{-\lambda w}}{(n-1)!}\right)}}.\end{array}$$ Let $N$ the number of arrivals within the fixed time interval. The pmf of $N$ is $$\begin{array}{cc}\mathbb{P}(N=n)& =\mathbb{P}(W\le t\cap \underset{V>t-W}{\underbrace{V+W>t}})\\ & =\underset{0}{\overset{t}{\int }}\underset{t-w}{\overset{\mathrm{\infty }}{\int }}\underset{\text{joint pdf of}(V,W)}{\underbrace{f(v,w)}}dvdw\\ & =\underset{0}{\overset{t}{\int }}\underset{t-w}{\overset{\mathrm{\infty }}{\int }}(\lambda e^{-\lambda v})\left(\frac{{\lambda }^n{w}^{n-1}{e}^{-\lambda w}}{(n-1)!}\right)dvdw\\ & =\underset{0}{\overset{t}{\int }}\frac{{\lambda }^n{w}^{n-1}{e}^{-\lambda w}}{(n-1)!}\underset{t-w}{\overset{\mathrm{\infty }}{\int }}\lambda e^{-\lambda v}dvdw\\ & =\frac{{\lambda }^n}{(n-1)!}\underset{0}{\overset{t}{\int }}{w}^{n-1}\cancel{e^{-\lambda w}}(0-(-e^{-\lambda (t\cancel{-w})}))dw\\ & =\frac{{\lambda }^n{e}^{-\lambda t}}{(n-1)!}\underset{0}{\overset{t}{\int }}{w}^{n-1}dw\\ & =\frac{{\lambda }^n{e}^{-\lambda t}}{(n-1)!}\cdot (\frac{t^n}{n}-0)\\ & =\frac{e^{-\lambda t}(\lambda t{)}^n}{n!}\end{array}$$ which is the pmf of $\mathrm{Pois}(\lambda t)$. The result follows.

$◻$

Proof. For each $t>0$, $$\begin{array}{cccc}& & \mathbb{P}(T>t)& =\mathbb{P}(T_1>t\cap \cdots \cap T_n>t)\\ & & & =\mathbb{P}(T_1>t)\cdots \mathbb{P}(T_n>t)& \text{by independence}\\ & & & =[1-(\underset{\text{cdf of}\mathrm{Exp}({\lambda }_1)}{\underbrace{1-e^{-{\lambda }_{1}t}}})]\cdots [1-(\underset{\text{cdf of}\mathrm{Exp}({\lambda }_n)}{\underbrace{1-e^{-{\lambda }_{n}t}}})]\\ & & & =e^{-t({\lambda }_1+\cdots +{\lambda }_n)}\\ & \Rightarrow & \mathbb{P}(T\le t)& =1-e^{-t({\lambda }_1+\cdots +{\lambda }_n)}\\ & \Rightarrow & T& \sim \mathrm{Exp}({\lambda }_1+{\lambda }_2+\cdots +{\lambda }_n)\end{array}$$

$◻$