Probability/Conditional Distributions

Motivation

Suppose there is an earthquake. Let $X$ be the number of casualties and $Y$ be the Richter scale of the earthquake.

(a) Without given anything, what is the distribution of $X$?

(b) Given that $Y=1$ , what is the distribution of $X$?

(c) Given that $Y=9$ , what is the distribution of $X$?

Template:Colored remark Are your answers to (a),(b),(c) different?

In (b) and (c), we have the Template:Colored em distribution of $X$ given $Y=1$, and the Template:Colored em distribution of $X$ given $Y=9$ respectively.

In general, we have Template:Colored em of $X$ given $Y$ (Template:Colored em observing the value of $Y$), or $X$ given $Y=y$ (Template:Colored em observing the value of $Y$).

Conditional distributions

Recall the definition of Template:Colored em: $$\mathbb{P}(A|B)=\frac{\mathbb{P}(A\cap B)}{\mathbb{P}(B)},$$ in which $A,B$ are events, with $\mathbb{P}(B)>0$. Applying this definition to Template:Colored em $X,Y$, we have $$\mathbb{P}(X=x|Y=y)=\frac{\mathbb{P}(X=x\cap Y=y)}{\mathbb{P}(Y=y)}=\frac{f(x,y)}{f_Y(y)},$$ where $f(x,y)$ is the joint pmf of $X$ and $Y$, and $f_Y(y)$ is the marginal pmf of $Y$. It is natural to call such conditional probability as Template:Colored em, right? We will denote such conditional probability as $f_{X|Y}(x|y)$. Then, this is basically the definition of Template:Colored em pmf: Template:Colored em pmf of $X$ given $Y=y$ is the conditional probability $\mathbb{P}(X=x|Y=y)$. Naturally, we will expect that Template:Colored em is defined similarly. This is indeed the case: Template:Colored definition Template:Colored remark To understand the definition more intuitively for the continuous case, consider the following diagram.

Top view:
     
        |
        |
        *---------------* 
        |               |
        |               |
fixed y *===============* <--- corresponding interval
        |               |
        |               |
        *---------------*
        |
        *---------------- x

Side view:

          *  
         / \ 
        *\  *  /                                           
       /|#\   \
   |  / |##\ / *---------*
   | *  |###\            /\
   | |\ |##/#\----------/--\     
   | | \|#/###*--------*   /                             
   | |  \/############/#\ /                              
   | |y *\===========/===*                               
   | | /  *---------*   /                                
   | |/              \ /                                 
   | *----------------*                                  
   |/                                                    
   *------------------------- x                          


Front view:
             
    |
    |
    |               
    *\     
    |#\    
    |##\   
    |###\             
    |####\   <------ Area: f_Y(y)
    |#####*--------*  
    |###############\ 
    *================*-------------- x

*---*
|###| : corresponding cross section from joint pdf
*---*   

We can see that when we are conditioning $Y=y$, we take a "slice" out from the region under joint pdf, and the area of the "whole slice" is the area between the Template:Colored em joint pdf $f(x,y)$ with fixed $y$ and variable $x$, and the $x$-axis. Since the area is given by $\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}f(x,y)dx=f_Y(y)$, while according to the probability axioms, the area should equal 1. Hence, we scale down the area of "slice" by a factor of $f_Y(y)$, by dividing the univariate joint pdf $f(x,y)$ by $f_Y(y)$. After that, the curve at the top of scaled "slice" is the graph of the conditional pdf $\frac{f(x,y)}{f_Y(y)}$.

Now, we have discussed the case where both random variables are discrete or continuous. How about the case where one of them is discrete and another one is continuous? In this case, there is no "joint probability function" of these two random variables, since one is discrete and another is continuous! But, we can still define the conditional probability function in some other ways. To motivate the following definition, let $F_{X|Y}(x|y)$ be the conditional probability $\mathbb{P}(X\le x|Y=y)$. Then, differentiating $F_{X|Y}(x|y)$ with respect to $x$ should yield the conditional pdf $f_{X|Y}(x|y)$. So, we have $$\begin{array}{cc}{f}_{X|Y}(x|y)=\frac{d}{dx}{F}_{X|Y}(x|y)& =\underset{h\to 0}{\lim }\frac{\mathbb{P}(X\le x+h|Y=y)-\mathbb{P}(X\le x|Y=y)}{h}\\ & =\underset{h\to 0}{\lim }\frac{\mathbb{P}(x<X\le x+h|Y=y)}{h}\\ & =\underset{h\to 0}{\lim }\frac{\mathbb{P}(Y=y|x<X\le x+h)\mathbb{P}(x<X\le x+h)}{h\mathbb{P}(Y=y)}\\ & =\underset{h\to 0}{\lim }\frac{\mathbb{P}(Y=y|x<X\le x+h)\mathbb{P}(x<X\le x+h)}{h\mathbb{P}(Y=y)}\\ & =\underset{h\to 0}{\lim }\frac{\mathbb{P}(Y=y|x\le X\le x+h)}{\mathbb{P}(Y=y)}\underset{h\to 0}{\lim }\frac{\mathbb{P}(x<X\le x+h)}{h}\\ & =\frac{\mathbb{P}(Y=y|X=x)\frac{d}{dx}{F}_X(x)}{\mathbb{P}(Y=y)}\\ & =\frac{\mathbb{P}(Y=y|X=x)f_X(x)}{\mathbb{P}(Y=y)}.\\ \end{array}$$ Thus, it is natural to have the following definition. Template:Colored definition Now, how about the case where $X$ is discrete and $Y$ is continuous? In this case, let us use the above definition for the motivation of definition. However, we should interchange $X$ and $Y$ so that the assumptions are still satisfied. Then, we get $$f_{Y|X}(y|x)=\frac{\mathbb{P}(X=x|Y=y)f_Y(y)}{\mathbb{P}(X=x)}.$$ In this case, $X$ is discrete, so it is natural to define the conditional pmf of $X$ given $Y=y$ as $\mathbb{P}(X=x|Y=y)$ in the expression. Now, after rearranging the terms, we get $$\mathbb{P}(X=x|Y=y)=\frac{f_{Y|X}(y|x)\mathbb{P}(X=x)}{f_Y(y)}.$$ Thus, we have the following definition. Template:Colored definition Based on the definitions of conditional probability functions, it is natural to define the Template:Colored em cdf as follows. Template:Colored definition Template:Colored remark Graphical illustration of the definition (continuous random variables):

Top view:
     
        |
        |
        *---------------* 
        |               |
        |               |
fixed y *=========@=====* <--- corresponding interval
        |         x     |
        |               |
        *---------------*
        |
        *---------------- 

Side view:

          *  
         / \ 
        *\  *  /                                           
       /|#\   \
   |  / |##\ / *---------*
   | *  |###\            /\
   | |\ |##/#\----------/--\     
   | | \|#/###*--------*   /                             
   | |  \/#########   / \ /                              
   | |y *\========@==/===*                               
   | | /  *-------x-*   /                                
   | |/              \ /                                 
   | *----------------*                                  
   |/                                                    
   *------------------------- x                          


Front view:

    |
    |
    |
    *\      
    |#\    
    |##\              
    |###\             
    |####\   <------------- Area: f_Y(y)         
    |#####*--------*  
    |###########    \ 
    *==========@=====*--------------  
               x
*---*
|###| : the desired region from the cross section from joint pdf, whose area is the probability from the cdf
*---*   

If $Y=\mathrm{𝟏}\{A\}$ for some event $A$, we have some special notations for simplicity:

• the conditional probability function of $X$ given $Y=y$ becomes

$$f_{X|Y}(x|y)=\{\begin{array}{cc}f(x|A),& y=1;\\ f(x|A^c),& y=0.\end{array}$$

• the conditional cdf of $X$ given $Y=y$ becomes

$$F_{X|Y}(x|y)=\mathbb{P}(X\le x|Y=y)=\{\begin{array}{cc}F(x|A),& y=1;\\ F(x|A^c),& y=0.\end{array}$$ Template:Colored proposition

Proof. Recall the definition of independence between two random variables:

$X,Y$ are independent if

$$f(x,y)=f_X(x)f_Y(y)$$

for each $x,y$.

Since $$f_{X|Y}(x|y)=\frac{\stackrel{f_X(x)f_Y(y)}{\overbrace{f(x,y)}}}{f_Y(y)}=f_X(x)\text{ and }{f}_{Y|X}(y|x)=\frac{\stackrel{f_Y(y)f_X(x)}{\overbrace{f(y,x)}}}{f_X(x)}=f_Y(y)$$ for each $x,y$, we have the desired result.

$◻$

We can extend the definition of conditional probability function and cdf to groups of random variables, for joint cdf's and joint probability functions, as follows: Template:Colored definition Then, we also have a similar proposition for determining independence of two random vectors. Template:Colored proposition

Proof. The definition of independence between two random vectors is

• $𝐗=(X_1,\dots ,X_r{)}^T,𝐘=(Y_1,\dots ,Y_s{)}^T$ are independent if

$$f(x_1,\dots ,x_r,y_1,\dots ,y_s)=f_{𝐗}(x_1,\dots ,x_r)f_{𝐘}(y_1,\dots ,y_s)$$

for each $x_1,\dots ,x_r,y_1,\dots ,y_s$.

Since $$f_{𝐗|𝐘}(x_1,\dots ,x_r|y_1,\dots ,y_s)=\frac{\stackrel{f_{𝐗}(x_1,\dots ,x_r)f_{𝐘}(y_1,\dots ,y_s)}{\overbrace{f(x_1,\dots ,x_r,y_1,\dots ,y_s)}}}{f_{𝐘}(y_1,\dots ,y_s)}=f_{𝐗}(x_1,\dots ,x_r)\text{ and }{f}_{𝐘|𝐗}(y_1,\dots ,y_s|x_1,\dots ,x_r)=\frac{\stackrel{f_{𝐘}(y_1,\dots ,y_s)f_{𝐗}(x_1,\dots ,x_r)}{\overbrace{f(y_1,\dots ,y_s,x_1,\dots ,x_r)}}}{f_{𝐗}(x_1,\dots ,x_r)}=f_{𝐘}(y_1,\dots ,y_s)$$ for each $x_1,\dots ,x_r,y_1,\dots ,y_s$, we have the desired result.

$◻$

Conditional distributions of bivariate normal distribution

Recall from the [[../Important Distributions]] chapter that the joint pdf of ${𝒩}_2(\mathit{\mu },\mathrm{\Sigma })$ is $$f(x,y)=\frac{1}{2\pi {\sigma }_X{\sigma }_Y\sqrt{1-{\rho }^2}}\exp (-\frac{1}{2(1-{\rho }^2)}({\left(\frac{x-{\mu }_X}{{\sigma }_X}\right)}^2-2\rho \left(\frac{x-{\mu }_X}{{\sigma }_X}\right)\left(\frac{y-{\mu }_Y}{{\sigma }_Y}\right)+{\left(\frac{y-{\mu }_Y}{{\sigma }_Y}\right)}^2)),(x,y)\in {ℝ}^2$$, and $X\sim 𝒩({\mu }_X,{\sigma }_X^2)$ and $Y\sim 𝒩({\mu }_Y,{\sigma }_Y^2)$ in this case. in which $\rho =\rho (X,Y)$ and ${\sigma }_X,{\sigma }_Y$ are positive. Template:Colored proposition

Proof.

• First, the conditional pdf

$$\begin{array}{cc}{f}_{X|Y}(x|y)& \stackrel{\text{ def }}{=}\frac{f(x,y)}{f_Y(y)}\\ & =\frac{1}{2\pi {\sigma }_X\cancel{{\sigma }_Y}\sqrt{1-{\rho }^2}}\exp (-\frac{1}{2(1-{\rho }^2)}({\left(\frac{x-{\mu }_X}{{\sigma }_X}\right)}^2-2\rho \left(\frac{x-{\mu }_X}{{\sigma }_X}\right)\left(\frac{y-{\mu }_Y}{{\sigma }_Y}\right)+{\left(\frac{y-{\mu }_Y}{{\sigma }_Y}\right)}^2))/\frac{1}{\sqrt{2\pi \cancel{{\sigma }_Y^2}}}\exp \left(-(y-{\mu }_Y{)}^2/2{\sigma }_Y^2\right)\\ & =\frac{1}{\sqrt{2\pi {\sigma }_X^2(1-{\rho }^2)}}\exp (-\frac{1}{2(1-{\rho }^2)}({\left(\frac{x-{\mu }_X}{{\sigma }_X}\right)}^2-2\rho \left(\frac{x-{\mu }_X}{{\sigma }_X}\right)\left(\frac{y-{\mu }_Y}{{\sigma }_Y}\right)+{\left(\frac{y-{\mu }_Y}{{\sigma }_Y}\right)}^2)+(y-{\mu }_Y{)}^2/2{\sigma }_Y^2)\\ & =\frac{1}{\sqrt{2\pi {\sigma }_X^2(1-{\rho }^2)}}\exp (-\frac{1}{2(1-{\rho }^2)}({\left(\frac{x-{\mu }_X}{{\sigma }_X}\right)}^2-2\rho \left(\frac{x-{\mu }_X}{{\sigma }_X}\right)\left(\frac{y-{\mu }_Y}{{\sigma }_Y}\right)+\cancel{{\left(\frac{y-{\mu }_Y}{{\sigma }_Y}\right)}^2}-(\cancel{1}-{\rho }^2){\left(\frac{y-{\mu }_Y}{{\sigma }_Y}\right)}^2))\\ & =\frac{1}{\sqrt{2\pi {\sigma }_X^2(1-{\rho }^2)}}\exp (-\frac{1}{2{\sigma }_X^2(1-{\rho }^2)}({(x-{\mu }_X)}^2-2\rho \cdot \frac{{\sigma }_X}{{\sigma }_Y}(x-{\mu }_X)(y-{\mu }_Y)+{(\rho \cdot \frac{{\sigma }_X}{{\sigma }_Y}(y-{\mu }_Y))}^2))\\ & =\frac{1}{\sqrt{2\pi {\sigma }_X^2(1-{\rho }^2)}}\exp (-\frac{1}{2{\sigma }_X^2(1-{\rho }^2)}{((x-{\mu }_X)-(\rho \cdot \frac{{\sigma }_X}{{\sigma }_Y}(y-{\mu }_Y)))}^2)\\ & =\frac{1}{\sqrt{2\pi {\sigma }_X^2(1-{\rho }^2)}}\exp (-\frac{1}{2{\sigma }_X^2(1-{\rho }^2)}{(x-{\mu }_X-\rho \cdot \frac{{\sigma }_X}{{\sigma }_Y}(y-{\mu }_Y))}^2)\end{array}$$

• Then, we can see that $X|(Y=y)\sim 𝒩({\mu }_X+\rho \cdot \frac{{\sigma }_X}{{\sigma }_Y}(y-{\mu }_Y),{\sigma }_X^2(1-{\rho }^2))$,
• and by symmetry (interchanging $X$ and $Y$, and also interchanging $x$ and $y$), $Y|(X=x)\sim 𝒩({\mu }_Y+\rho \cdot \frac{{\sigma }_Y}{{\sigma }_X}(x-{\mu }_X),{\sigma }_Y^2(1-{\rho }^2))$.

$◻$

Conditional version of concepts

We can obtain Template:Colored em version of concepts previously established for 'unconditional' distributions analogously for Template:Colored em distributions by substituting 'unconditional' cdf, pdf or pmf, i.e. $F(\cdot )$ or $f(\cdot )$, by their Template:Colored em counterparts, i.e. $F(\cdot |\cdot )$ or $f(\cdot |\cdot )$.

Conditional expectation

Template:Colored definition Template:Colored remark Similarly, we have conditional version of law of the unconscious statistician. Template:Colored proposition Template:Colored proposition

Proof. $$𝔼[g(X)|Y]=\{\begin{array}{cc}\sum _x^{}g(x)f_{X|Y}(x|Y)=\sum _x^{}g(x)f_X(x)=𝔼[g(X)],& X\text{ is discrete};\\ \underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}g(x)f_{X|Y}(x|Y)dx=\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}g(x)f_X(x)dx=𝔼[g(X)],& X\text{ is continuous}.\end{array}$$

$◻$

Template:Colored remark Template:Colored example The properties of $𝔼[\cdot ]$ still hold for conditional expectations $𝔼[\cdot |Y]$, with Template:Colored em 'unconditional' expectation replaced by Template:Colored em expectation and some suitable modifications, as follows: Template:Colored proposition

Proof. The proof is similar to the one for 'unconditional' expectations.

$◻$

Template:Colored remark The following theorem about conditional expectation is quite important. Template:Colored theorem

Proof. $$𝔼[𝔼[g(X)|Y]]=\{\begin{array}{cc}\sum _y^{}𝔼[g(X)|Y=y]f_Y(y)=\sum _x^{}(\sum _y^{}g(x)\stackrel{f(x,y)\cancel{/f_Y(y)}}{\overbrace{f_{X|Y}(x|y)}}\cancel{f_Y(y)})=\sum _x^{}g(x)(\stackrel{f_X(x)}{\overbrace{\sum _y^{}f(x,y)}})=𝔼[g(X)],& X\text{ is discrete};\\ \underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}𝔼[g(X)|Y=y]f_Y(y)dy=\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}(\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}g(x)\underset{f(x,y)\cancel{/f_Y(y)}}{\underbrace{f_{X|Y}(x|y)}}dx)\cancel{f_Y(y)}dy=\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}g(x)(\underset{f_X(x)}{\underbrace{\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}f(x,y)dy}})dx=𝔼[g(X)],& X\text{ is continuous}.\end{array}$$

$◻$

Proof.

• First,

$$𝔼[\mathrm{𝟏}\{A\}|Y]=1(\mathbb{P}(\mathrm{𝟏}\{A\}=1|Y)+0(\mathbb{P}(\mathrm{𝟏}\{A\}=0|Y)=\mathbb{P}(A|Y).$$

• Then, using law of total expectation,

$${𝔼}_Y[\mathbb{P}(A|Y)]\stackrel{\text{ above }}{=}{𝔼}_Y[𝔼[\mathrm{𝟏}\{A\}|Y]]=𝔼[\mathrm{𝟏}\{A\}]=\mathbb{P}(A).$$

$◻$

Proof. Define $Y=i$ if $A_i$ occurs, in which $i$ is a positive integer. Then, $$𝔼[X]={𝔼}_Y[{𝔼}_X[X|Y]]=\sum _{i=1}^{\mathrm{\infty }}{𝔼}_X[X|Y=i]\mathbb{P}(Y=i)=\sum _{i=1}^{\mathrm{\infty }}𝔼[X|A_i]\mathbb{P}(A_i)$$

$◻$

Proof. By the formula of expectation computed by weighted average of conditional expectations, $$𝔼[X\mathrm{𝟏}\{A\}]=𝔼[X\underset{1}{\underbrace{\mathrm{𝟏}\{A\}}}|A]\mathbb{P}(A)+𝔼[X\underset{0}{\underbrace{\mathrm{𝟏}\{A\}}}|A^c]\mathbb{P}(A^c)=𝔼[X|A]\mathbb{P}(A),$$ and the result follows if $\mathbb{P}(A)>0$.

$◻$

Template:Colored remark After defining Template:Colored em expectation, we can also have Template:Colored em variance, covariance and correlation coefficient, since variance, covariance, and correlation coefficient are built upon expectation.

Conditional expectations of bivariate normal distribution

Proof.

• The result follows from the proposition about conditional distributions of bivariate normal distribution readily.

$◻$

Conditional variance

Template:Colored definition Similarly, we have properties of Template:Colored em variance which are similar to that of variance. Template:Colored proposition

Proof. The proof is similar to the one for properties of variance.

$◻$

Beside law of total expectation, we also have law of total variance, as follows: Template:Colored proposition

Proof. $$\begin{array}{cc}𝔼[\mathrm{Var}(X|Y)]+\mathrm{Var}(𝔼[X|Y])& =𝔼[𝔼[X^2|Y]-(𝔼[X|Y]{)}^2]+𝔼[(𝔼[X|Y]{)}^2]-(𝔼[𝔼[X|Y]]{)}^2\\ & =𝔼[𝔼[X^2|Y]]\cancel{+𝔼[(𝔼[X|Y]{)}^2]}+𝔼[(𝔼[X|Y]{)}^2]\cancel{-(𝔼[𝔼[X|Y]]{)}^2}\\ & =𝔼[X^2]-(𝔼[X]{)}^2\text{by law of total expectation}\\ & =\mathrm{Var}(X)\end{array}$$

$◻$

Conditional variances of bivariate normal distribution

Proof.

• The result follows from he proposition about conditional distributions of bivariate normal distribution readily.

$◻$