Probability/Transformation of Random Variables

Transformation of random variables

Underlying principle

Let $X_1,\dots ,X_n$ be $n$ random variables, $Y_1,\dots ,Y_n$ be another $n$ random variables, and $𝐗=(X_1,\dots ,X_n{)}^T,𝐘=(Y_1,\dots ,Y_n{)}^T$ be random (column) vectors.

Suppose the vector-valued function $𝐠:\mathrm{supp}(𝐗)\to \mathrm{supp}(𝐘)$ is bijective (it is also called one-to-one correspondence in this case). Then, its inverse ${𝐠}^{-1}:\mathrm{supp}(𝐘)\to \mathrm{supp}(𝐗)$ exists.

After that, we can transform $𝐗$ to $𝐘$ by applying the transformation $𝐠$, i.e. by $𝐘=𝐠(𝐗)$, and transform $𝐘$ to $𝐗$ by applying the inverse transformation ${𝐠}^{-1}$, i.e. by $𝐗={𝐠}^{-1}(𝐘)$.

We are often interested in deriving the joint probability function $f_{𝐘}(𝐲)$ of $𝐘$, given the joint probability function $f_{𝐗}(𝐱)$ of $𝐗$. We will examine the Template:Colored em and Template:Colored em cases one by one in the following.

Transformation of discrete random variables

Proof. Considering the original pmf $f_{𝐘}(𝐲)$, we have $$f_{𝐘}(𝐲)\stackrel{\text{ def }}{=}\mathbb{P}(𝐘=𝐲)=\mathbb{P}({𝐠}^{-1}(𝐘)={𝐠}^{-1}(𝐲))=\mathbb{P}(𝐗={𝐠}^{-1}(𝐲))\stackrel{\text{ def }}{=}{f}_{𝐗}({𝐠}^{-1}(𝐲)),𝐲\in \mathrm{supp}(𝐘).$$ In particular, the inverse ${𝐠}^{-1}$ exists since $𝐠$ is bijective.

$◻$

Transformation of continuous random variables

For Template:Colored em random variables, the situation is more complicated.

Let us investigate the case for univariate pdf, which is simpler. Template:Colored theorem

Proof. Under the assumption that $g$ is differentiable and strictly monotone, the cdf $F_Y(y)=\mathbb{P}(g(X)\le y)=\{\begin{array}{cc}\mathbb{P}(X\le g^{-1}(y))=F_X(g^{-1}(y)),& g^{-1}\text{ is increasing};\\ \mathbb{P}(X\ge g^{-1}(y))=1-F_X(g^{-1}(y)),& g^{-1}\text{ is decreasing}.\end{array}$ ($g^{-1}$ exists since $g$ is strictly monotonic.) Differentiating both side of the above equation (assuming the cdf's involved are differentiable) gives $$f_Y(y)=\{\begin{array}{cc}{f}_X(g^{-1}(y))\frac{d{g}^{-1}(y)}{dy},& g^{-1}\text{ is increasing};\\ -f_X(g^{-1}(y))\frac{d{g}^{-1}(y)}{dy},& g^{-1}\text{ is decreasing}.\end{array}$$ Since $x=g^{-1}(y)$, we can write $\frac{d{g}^{-1}(y)}{dy}$ as $\frac{dx}{dy}$. Also, we can summarize the above case defined function into a single expression by applying absolute value function to both side: $$f_Y(y)=f_X(g^{-1}(y))\left|\frac{dx}{dy}\right|,$$ where the absolute value sign is only applied to $\frac{dx}{dy}$ since the pdf's must be nonnegative, and thus we do not need to apply the sign to them.

$◻$

Template:Colored remark Let us define Template:Colored em, and introduce several notations in the definition. Template:Colored definition Template:Colored remark Template:Colored example

Proof. Template:Colored em: Assume $𝐠$ is differentiable and bijective.

First, $$\mathbb{P}(Y\in S)=\underset{}{\overset{}{\int }}\cdots \underset{S}{\overset{}{\int }}{f}_{𝐘}(𝐲)d{y}_1\cdots d{y}_n(1).$$

On the other hand, we have $$\mathbb{P}(Y\in S)=\mathbb{P}(X={𝐠}^{-1}(Y)\in {𝐠}^{-1}(S))=\int \cdots \underset{{𝐠}^{-1}(S)}{\overset{}{\int }}{f}_{𝐗}(𝐱)d{x}_1\cdots d{x}_n$$ where ${𝐠}^{-1}(S)=\{x\in X:𝐠(x)\in S\}$, which is the preimage of the set $S$ under $𝐠$.

Applying the change of variable formula to this integral (whose proof is advanced and uses our assumptions), we get $$\int \cdots \underset{{𝐠}^{-1}(S)}{\int }{f}_{𝐗}(𝐱)d{x}_1\cdots d{x}_n=\int \cdots \underset{S}{\int }{f}_{𝐗}({𝐠}^{-1}(𝐲))|\det \frac{\partial 𝐱}{\partial 𝐲}|d{y}_1\cdots d{y}_n(2)$$ Comparing the integrals in $(1)$ and $(2)$, we can observe the desired result.

$◻$

Moment generating function

Proof.

• Since

$$M_X(t)=𝔼\left[e^{tX}\right]=𝔼[1+tX+\frac{t^2{X}^2}{2!}+\cdots ]\stackrel{\text{linearity}}{=}1+t𝔼[X]+\frac{t^2}{2!}𝔼[X^2]+\cdots ,$$ $$\frac{d^n}{d{t}^n}{M}_X(t){|}_{t=0}=\frac{d^n}{d{t}^n}(1+t𝔼[X]+\frac{t^2}{2!}𝔼[X^2]+\cdots ){|}_{t=0}=𝔼[X]\frac{d^n}{d{t}^n}t+\frac{𝔼[X^2]}{2!}\frac{d^n}{d{t}^n}{t}^2+\cdots ,$$

• The result follows from simplifying the above expression by $\frac{d^n}{d{t}^n}{t}^m=\mathrm{𝟏}\{m=n\}n!+\mathrm{𝟏}\{m\ne n\}(0).$

$◻$

Proof. $$M_{XY}(t)=𝔼[e^{tXY}]\stackrel{\text{lote}}{=}{𝔼}_X[{{𝔼}_Y[e}^{tXY}|X]]={𝔼}_X[M_Y(tX)].$$ Similarly, $$M_{XY}(t)=𝔼[e^{tXY}]\stackrel{\text{lote}}{=}{𝔼}_X[{{𝔼}_X[e}^{tXY}|Y]]={𝔼}_Y[M_X(tY)].$$

• lote: law of total expectation

$◻$

Joint moment generating function

In the following, we will use $𝐗$ to denote $(X_1,\dots ,X_n{)}^T$. Template:Colored definition Template:Colored remark Template:Colored proposition

Proof. 'only if' part: Assume $X_1,\dots ,X_n$ are independent. Then, $$M_{𝐗}(𝐭)=𝔼[e^{𝐭\cdot 𝐗}]=𝔼[e^{t_1{X}_1}\cdots e^{t_n{X}_n}]\stackrel{\text{ independence }}{=}𝔼[e^{t_1{X}_1}]\cdots 𝔼[e^{t_n{X}_n}]=M_{X_1}(t_1)\cdots M_{X_n}(t_n).$$ Proof for 'if' part is quite complicated, and thus is omitted.

$◻$

Analogously, we have Template:Colored em mgf. Template:Colored definition Template:Colored proposition

Proof. $$M_{𝐚\cdot 𝐗+b}(t)=𝔼[e^{t𝐚\cdot 𝐗+bt}]=e^{bt}𝔼[e^{t𝐚\cdot 𝐗}]=e^{bt}{M}_{𝐗}(t𝐚)=e^{bt}{M}_{𝐗}(t{a}_1,\dots ,t{a}_n).$$

$◻$

Moment generating function of some important distributions

Proof. $$M_X(t)=\sum _{k=0}^n{e}^{tk}\underset{\text{for}\mathrm{Binom}(n,p)}{\underbrace{(\genfrac{}{}{0ex}{}{n}{k})p^k(1-p{)}^{n-k}}}=\sum _{k=0}^n(\genfrac{}{}{0ex}{}{n}{k})(p{e}^t{)}^k(1-p{)}^{n-k}=(p{e}^t+1-p{)}^n\text{by binomial theorem}.$$

$◻$

Proof. $$M_X(t)\stackrel{\text{ def }}{=}𝔼[e^{tX}]\stackrel{\text{ LOTUS }}{=}\sum _{k=0}^{\mathrm{\infty }}{e}^{tk}\cdot \underset{\text{for}\mathrm{Pois}(\lambda )}{\underbrace{\frac{e^{-\lambda }{\lambda }^k}{k!}}}=e^{\lambda (e^t-1)}\stackrel{=1}{\overbrace{\sum _{k=0}^{\mathrm{\infty }}\underset{\text{for}\mathrm{Pois}(\lambda e^t)}{\underbrace{\frac{e^{-\lambda e^t}(\lambda e^t{)}^k}{k!}}}}}=e^{\lambda (e^t-1)}.$$

$◻$

Proof.

• $$M_X(t)=𝔼[e^{tX}]=\lambda \underset{0}{\overset{\mathrm{\infty }}{\int }}{e}^{tx}{e}^{-\lambda x}dx=\lambda \underset{0}{\overset{\mathrm{\infty }}{\int }}{e}^{-(\lambda -t)x}dx=\frac{\lambda }{\lambda -t}\stackrel{=1}{\overbrace{\underset{0}{\overset{\mathrm{\infty }}{\int }}\underset{\text{for}\mathrm{Exp}(\lambda -t)}{\underbrace{(\lambda -t)e^{-(\lambda -t)x}}}dx}},\underset{\text{ensuring valid rate parameter}}{\underbrace{\lambda -t>0}}\iff t<\lambda .$$
• The result follows.

$◻$

Proof.

• We use similar proof technique from the proof for mgf of exponential distribution.

$$M_X(t)=𝔼[e^{tX}]=\frac{{\lambda }^{\alpha }}{\mathrm{\Gamma }(\alpha )}\underset{0}{\overset{\mathrm{\infty }}{\int }}{e}^{tx}{x}^{\alpha -1}{e}^{-\lambda x}dx=\frac{{\lambda }^{\alpha }}{\mathrm{\Gamma }(\alpha )}\underset{0}{\overset{\mathrm{\infty }}{\int }}{e}^{-(\lambda -t)x}{x}^{\alpha -1}dx=\frac{{\lambda }^{\alpha }}{(\lambda -t{)}^{\alpha }}\stackrel{=1}{\overbrace{\underset{0}{\overset{\mathrm{\infty }}{\int }}\underset{\text{for}\mathrm{Gamma}(\alpha ,\lambda -t)}{\underbrace{\frac{(\lambda -t{)}^{\alpha }}{\mathrm{\Gamma }(\alpha )}{e}^{-(\lambda -t)x}{x}^{\alpha -1}}}dx}},\underset{\text{ensuring valid rate parameter}}{\underbrace{\lambda -t>0}}\iff t<\lambda .$$

• The result follows.

$◻$

Proof.

• Let $Z=\frac{X-\mu }{\sigma }\sim 𝒩(0,1)$. Then, $X=\sigma Z+\mu$.
• First, consider the mgf of $Z$:

$$M_Z(t)\stackrel{\text{ def }}{=}𝔼[e^{tZ}]=\frac{1}{\sqrt{2\pi }}\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\underset{=e^{-(x^2-2tx)/2}}{\underbrace{e^{tx}{e}^{-x^2/2}}}dx=\frac{1}{\sqrt{2\pi }}\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\exp (\stackrel{=-(x-t{)}^2}{\overbrace{-(x^2-2tx+t^2)}}/2+t^2/2)dx=e^{t^2/2}\stackrel{=1}{\overbrace{\underset{-\mathrm{\infty }}{\overset{\mathrm{\infty }}{\int }}\underset{\text{for }𝒩(t,1)}{\underbrace{\frac{1}{\sqrt{2\pi }}\cdot e^{-(x-t{)}^2/2}}}dx}}=e^{t^2/2}.$$

• It follows that the mgf of $X$ is

$$M_X(t)=e^{\mu t}{M}_X(\sigma t)=e^{\mu t}{e}^{{\sigma }^2{t}^2/2}.$$

• The result follows.

$◻$

Distribution of linear transformation of random variables

We will prove some propositions about distributions of linear transformation of random variables using Template:Colored em. Some of them are mentioned in previous chapters. As we will see, proving these propositions using mgf is quite simple. Template:Colored proposition

Proof.

• The mgf of $aX+b$ is

$$M_{aX+b}(t)=e^{bt}{M}_X(at)=e^{bt}(\exp (a\mu t+(a\sigma {)}^2{t}^2/2))=\exp ((a\mu +b)t+a^2{\sigma }^2{t}^2/2),$$

which is the mgf of $𝒩(a\mu +b,a^2{\sigma }^2)$, and the result follows since mgf identify a distribution uniquely.

$◻$

Sum of independent random variables

Proof.

• The mgf of $X_1+\cdots +X_n$ is

$$M_{X_1+\cdots +X_n}(t)=M_{X_1}(t)\cdots M_{X_n}(t)=(p{e}^t+1-p{)}^{n_1}\cdots (p{e}^t+1-p{)}^{n_m}=(p{e}^t+1-p{)}^{n_1+\cdots +n_m},$$

which is the mgf of $\mathrm{Binom}(n_1+\cdots +n_m,p)$, as desired.

$◻$

Proof.

• The mgf of $X_1+\cdots +X_n$ is

$$M_{X_1+\cdots +X_n}(t)=M_{X_1}(t)\cdots M_{X_n}(t)=e^{{\lambda }_1(e^t-1)}\cdots e^{{\lambda }_n(e^t-1)}=e^{({\lambda }_1+\cdots +{\lambda }_n)(e^t-1)},$$

which is the mgf of $\mathrm{Pois}({\lambda }_1+\cdots +{\lambda }_n)$, as desired.

$◻$

Proof.

• The mgf of $X_1+\cdots +X_n$ is

$$M_{X_1+\cdots +X_n}(t)=M_{X_1}(t)\cdots M_{X_n}(t)={\left(\frac{\lambda }{\lambda -t}\right)}^n,$$

which is the mgf of $\mathrm{Gamma}(n,\lambda )$, as desired.

$◻$

Proof.

• The mgf of $X_1+\cdots +X_n$ is

$$M_{X_1+\cdots +X_n}(t)=M_{X_1}(t)\cdots M_{X_n}(t)={\left(\frac{\lambda }{\lambda -t}\right)}^{{\alpha }_1}\cdots {\left(\frac{\lambda }{\lambda -t}\right)}^{{\alpha }_n}={\left(\frac{\lambda }{\lambda -t}\right)}^{{\alpha }_1+\cdots +{\alpha }_n},$$

which is the mgf of $\mathrm{Gamma}({\alpha }_1+\cdots +{\alpha }_n,\lambda )$, as desired.

$◻$

Proof.

• The mgf of $X_1+\cdots +X_n$ (in which they are independent) is

$$M_{X_1+\cdots +X_n}(t)=M_{X_1}(t)\cdots M_{X_n}(t)=\exp ({\mu }_{1}t+{\sigma }_1^2{t}^2/2)\cdots \exp ({\mu }_{n}t+{\sigma }_n^2{t}^2/2)=\exp (({\mu }_1+\cdots +{\mu }_n)t+({\sigma }_1^2+\cdots +{\sigma }_n^2)t^2/2),$$

which is the mgf of $𝒩({\mu }_1+\cdots +{\mu }_n,{\sigma }_1^2+\cdots +{\sigma }_n^2)$, as desired.

$◻$

Central limit theorem

We will provide a proof to Template:Colored em (CLT) using mgf here. Template:Colored theorem

Proof.

• Define $T_n=\frac{\sqrt{n}({\bar{X}}_n-\mu )}{\sigma }$. Then, we have

$$T_n=\frac{\sqrt{n}((X_1+\cdots +X_n)/n-\mu )}{\sigma }=\frac{X_1+\cdots +X_n}{\sigma \sqrt{n}}-\frac{\sqrt{n}\mu }{\sigma },$$

• which is in the form of $𝐚\cdot 𝐗+b,𝐚={(\frac{1}{\sigma \sqrt{n}},\dots ,\frac{1}{\sigma \sqrt{n}})}^T\text{ and }b=-\frac{\sqrt{n}\mu }{\sigma }$.
• Therefore,

$$\begin{array}{cccc}& & M_{T_n}(t)& =e^{-\sqrt{n}\mu t/\sigma }(M_{X_1}\left(\frac{t}{\sigma \sqrt{n}}\right)\cdots M_{X_n}\left(\frac{t}{\sigma \sqrt{n}}\right))\\ & & M_{T_n}(t)& =e^{-\sqrt{n}\mu t/\sigma }{\left(M_{X_1}\left(\frac{t}{\sigma \sqrt{n}}\right)\right)}^n\text{since }{X}_1,\dots ,X_n\text{ are identically distributed, which is equivalent to they have the same mgf}\\ & \Rightarrow & \ln M_{T_n}(t)& =-\sqrt{n}\mu t/\sigma +n\ln (𝔼[e^{t/(\sigma \sqrt{n})X_1}])\\ & & & =-\sqrt{n}\mu t/\sigma +n\ln 𝔼[1+t/(\sigma \sqrt{n})X_1+(1/2!)t^2/({\sigma }^{2}n)+\cdots ]\text{since }{e}^x=1+x+\frac{x^2}{2!}+\cdots \\ & & & =-\sqrt{n}\mu t/\sigma +n\ln (1+t/(\sigma \sqrt{n})𝔼[X]+(1/2!)t^2/({\sigma }^{2}n)(\underset{\mathrm{Var}(X)+(𝔼[X]{)}^2}{\underbrace{𝔼[X^2]}})+\text{terms of order smaller than }{n}^{-1})\\ & & & =-\sqrt{n}\mu t/\sigma +n\ln (1+t/(\sigma \sqrt{n})\mu +(1/2)t^2/({\sigma }^{2}n)({\sigma }^2+{\mu }^2)+\text{terms of order smaller than }{n}^{-1})\\ & & & =-\sqrt{n}\mu t/\sigma +n[t/(\sigma \sqrt{n})\mu +(1/2)t^2/({\sigma }^{2}n)({\sigma }^2+{\mu }^2)-(1/2)(t/(\sigma \sqrt{n})\mu {)}^2+\text{terms of order smaller than }{n}^{-1}]\text{since }\ln (1+x)=x-x^2/2+\cdots \\ & & & =\cancel{-\sqrt{n}\mu t/\sigma }\cancel{+\sqrt{n}\mu t/\sigma }+\cancel{n}(1/2)t^2/({\sigma }^2\cancel{n})({\sigma }^2+{\mu }^2)-\cancel{n}(1/2)(t^2/({\sigma }^2\cancel{n}){\mu }^2)+\text{terms of order smaller than }{n}^0\\ & & & =(1/2)t^2({\sigma }^2/{\sigma }^2)\cancel{+(1/2){\mu }^2{t}^2-(1/2)t^2({\mu }^2)}+\text{terms of order smaller than }{n}^0\\ & & & =(1/2)t^2+\underset{\to 0\text{ as }n\to \mathrm{\infty }}{\underbrace{\text{terms of order smaller than }{n}^0}}\\ & \Rightarrow & \underset{n\to \mathrm{\infty }}{\lim }{M}_{T_n}(t)& =\underset{\text{mgf of }𝒩(0,1)}{\underbrace{e^{t^2/2}}},\end{array}$$ and the result follows from the mgf property of identifying distribution uniquely.

$◻$

Template:Colored remark A special case of using CLT as Template:Colored em is using normal distribution to Template:Colored em discrete distribution. To improve accuracy, we should ideally have Template:Colored em, as explained in the following. Template:Colored proposition Template:Colored remark Illustration of continuity correcction:

| 
|              /
|             /
|            /
|           /|
|          /#|
|         *##|
|        /|##|
|       /#|##|   
|      /##|##|   
|     /|##|##|   
|    / |##|##|   
|   /  |##|##|
|  /   |##|##|
| /    |##|##|
*------*--*--*---------------------
    i-1/2 i i+1/2

| 
|              /
|             /
|            /
|           / 
|          /  
|         *   
|        /|   
|       /#|      
|      /##|      
|     /###|      
|    /####|      
|   /#####|   
|  /|#####|   
| / |#####|   
*---*-----*------------------------
   i-1    i      

| 
|              /|
|             /#|
|            /##|
|           /###|
|          /####|
|         *#####|
|        /|#####|
|       / |#####|
|      /  |#####|
|     /   |#####|
|    /    |#####|
|   /     |#####| 
|  /      |#####|
| /       |#####|
*---------*-----*------------------
          i     i+1