Engineering Analysis/Linear Independence and Basis

Linear Independence

A set of vectors $V=v_1,v_2,\cdots ,v_n$ are said to be linearly dependent on one another if any vector v from the set can be constructed from a linear combination of the other vectors in the set. Given the following linear equation:

$a_1{v}_1+a_2{v}_2+\cdots +a_n{v}_n=0$

The set of vectors V is linearly independent only if all the a coefficients are zero. If we combine the v vectors together into a single row vector:

$\hat{V}=[v_1{v}_2\cdots v_n]$

And we combine all the a coefficients into a single column vector:

$\hat{a}=[a_1{a}_2\cdots a_n{]}^T$

We have the following linear equation:

$\hat{V}\hat{a}=0$

We can show that this equation can only be satisifed for $\hat{a}=0$, the matrix $\hat{V}$ must be invertable:

${\hat{V}}^{-1}\hat{V}\hat{a}={\hat{V}}^{-1}0$

$\hat{a}=0$

Remember that for the matrix to be invertable, the determinate must be non-zero.

Non-Square Matrix V

If the matrix $\hat{V}$ is not square, then the determinate can not be taken, and therefore the matrix is not invertable. To solve this problem, we can premultiply by the transpose matrix:

${\hat{V}}^T\hat{V}\hat{a}=0$

And then the square matrix ${\hat{V}}^T\hat{V}$ must be invertable:

$({\hat{V}}^T\hat{V}{)}^{-1}{\hat{V}}^T\hat{V}\hat{a}=0$

$\hat{a}=0$

Rank

The rank of a matrix is the largest number of linearly independent rows or columns in the matrix.

To determine the Rank, typically the matrix is reduced to row-echelon form. From the reduced form, the number of non-zero rows, or the number of non-zero columns (whichever is smaller) is the rank of the matrix.

If we multiply two matrices A and B, and the result is C:

$AB=C$

Then the rank of C is the minimum value between the ranks A and B:

$\mathrm{Rank}(C)=\min [\mathrm{Rank}(A),\mathrm{Rank}(B)]$

Span

A Span of a set of vectors V is the set of all vectors that can be created by a linear combination of the vectors.

Basis

A basis is a set of linearly-independent vectors that span the entire vector space.

Basis Expansion

If we have a vector $y\in V$, and V has basis vectors $v_1{v}_2\cdots v_n$, by definition, we can write y in terms of a linear combination of the basis vectors:

$a_1{v}_1+a_2{v}_2+\cdots +a_n{v}_n=y$

or

$\hat{V}\hat{a}=y$

If $\hat{V}$ is invertable, the answer is apparent, but if $\hat{V}$ is not invertable, then we can perform the following technique:

${\hat{V}}^T\hat{V}\hat{a}={\hat{V}}^{T}y$

$\hat{a}=({\hat{V}}^T\hat{V}{)}^{-1}{\hat{V}}^{T}y$

And we call the quantity $({\hat{V}}^T\hat{V}{)}^{-1}{\hat{V}}^T$ the left-pseudoinverse of $\hat{V}$.

Change of Basis

Frequently, it is useful to change the basis vectors to a different set of vectors that span the set, but have different properties. If we have a space V, with basis vectors $\hat{V}$ and a vector in V called x, we can use the new basis vectors $\hat{W}$ to represent x:

$x=\sum _{i=0}^n{a}_i{v}_i=\sum _{j=1}^n{b}_j{w}_j$

or,

$x=\hat{V}\hat{a}=\hat{W}\hat{b}$

If V is invertable, then the solution to this problem is simple.

Grahm-Schmidt Orthogonalization

If we have a set of basis vectors that are not orthogonal, we can use a process known as orthogonalization to produce a new set of basis vectors for the same space that are orthogonal:

Given: $\hat{V}=x_1{v}_2\cdots v_n$

Find the new basis $\hat{W}=w_1{w}_2\cdots w_n$

Such that $⟨w_i,w_j⟩=0\mathrm{\forall }i,j$

We can define the vectors as follows:

1. $w_1=v_1$
2. $w_m=v_m-\sum _{i=1}^{m-1}\frac{⟨v_m,u_i⟩}{⟨u_i,u_i⟩}{u}_i$

Notice that the vectors produced by this technique are orthogonal to each other, but they are not necessarily orthonormal. To make the w vectors orthonormal, you must divide each one by its norm:

$\overline{w}=\frac{w}{\Vert w\Vert }$

Reciprocal Basis

A Reciprocal basis is a special type of basis that is related to the original basis. The reciprocal basis $\hat{W}$ can be defined as:

$\hat{W}=[{\hat{V}}^T{]}^{-1}$