Matrix Operations

Matrix Multiplication

Matrix multiplication involves a series of vector dot products. To review matrix multiplication, see the below example of a vector being multiplied by a matrix.

$$ \begin{bmatrix} 1 & 2\\ 2 & 6 \end{bmatrix} \begin{bmatrix} 3\\ 5 \end{bmatrix} = \begin{bmatrix} 13\\ 36 \end{bmatrix} $$

This matrix multiplication can be rewritten to show the two dot products performed.

$$ \begin{bmatrix} 1 & 2 \end{bmatrix} \begin{bmatrix} 3\\ 5 \end{bmatrix} = 13; \begin{bmatrix} 2 & 6 \end{bmatrix} \begin{bmatrix} 3\\ 5 \end{bmatrix} = 36; \xrightarrow{} \begin{bmatrix} 1(3)+2(5)\\ 2(3)+6(5) \end{bmatrix} = \begin{bmatrix} 13\\ 36 \end{bmatrix} $$

An easy way to remember how to perform matrix multiplication is by remembering to multiply across a row and down a column. The first row vector gets multiplied with the column vector as a vector dot product. This is the first variable in the new vector or matrix. Any vector product of the first row in the operating matrix will end up in the first row of the new vector or matrix. Similarly with the second and further rows.

Another important consideration to keep in mind for any matrix multiplication is dimensional compatibility. If two matrices or vectors do not have matching dimensions for dot products, it is not possible to multiply. The number of components in a row of an operating matrix needs to match the number of components of the columns of the vector or matrix being operated on. Consider the following matrices and their dimensions.

$$ A = \begin{bmatrix} 2 & 2\\ 3 & 5 \end{bmatrix} $$ \[ \text{dim}(A) = 2 \times 2 \]

$$ B = \begin{bmatrix} 3 & 5 & 5 & 1\\ 4 & 4 & 2 & 6 \end{bmatrix} $$ \[ \text{dim}(B) = 2 \times 4 \]

$$ C = \begin{bmatrix} 3 & 4\\ 1 & 1\\ 6 & 7 \end{bmatrix} $$ \[ \text{dim}(C) = 3 \times 2 \]

The dimensions of each matrix is written as $ \textit{number of rows} \times \textit{number of columns} $ or $ \textit{m} \times \textit{n} $. It is possible to determine if two matrices can be multiplied together and the dimensions of the resultant matrix by comparing the dimensions of two matrices. Before multiplying the above matrices, we will compare the dimensionality of each potential product.

\[ A \times B \to (2 \times 2)(2 \times 4) \to (2 \times 4) \] \[ B \times A \to (2 \times 4)(2 \times 2) \to \text{Not Possible} \]

To test for compatibility, compare the inner terms of the dimensions. In the first example, the number of columns in $ \textit{A} $ matched the number of rows in $ \textit{B} $. The dimensions of the new matrix will become the number of rows in the operating matrix $ \textit{A} $ by the number of columns in the matrix being operated on $ \textit{B} $.

Matrix Multiplication Examples

The following examples will use the given matrices of $ \textit{A, B,} $ and $ \textit{C} $ to compute matrix products. Use these examples to practice multiplying rectangular matrices.

$$ A \times B = \begin{bmatrix} 2 & 2\\ 3 & 5 \end{bmatrix} \begin{bmatrix} 3 & 5 & 5 & 1\\ 4 & 4 & 2 & 6 \end{bmatrix} = $$

$$ \begin{bmatrix} 2(3)+2(4) & 2(5)+2(4) & 2(5)+2(2) & 2(1)+2(6)\\ 3(3)+5(4) & 3(5)+5(4) & 3(5)+5(2) & 3(1)+5(6) \end{bmatrix} = \begin{bmatrix} 14 & 18 & 14 & 14\\ 29 & 35 & 25 & 33 \end{bmatrix} $$

$$ C \times A = \begin{bmatrix} 3 & 4\\ 1 & 1\\ 6 & 7 \end{bmatrix} \begin{bmatrix} 2 & 2\\ 3 & 5 \end{bmatrix} = $$

$$ \begin{bmatrix} 3(2)+4(3) & 3(2)+4(5)\\ 1(2)+1(3) & 1(2)+1(5)\\ 6(2)+7(3) & 6(2)+7(5) \end{bmatrix} = \begin{bmatrix} 18 & 26\\ 5 & 7\\ 33 & 47 \end{bmatrix} $$

$$ C \times B = \begin{bmatrix} 3 & 4\\ 1 & 1\\ 6 & 7 \end{bmatrix} \begin{bmatrix} 3 & 5 & 5 & 1\\ 4 & 4 & 2 & 6 \end{bmatrix} = $$

$$ \begin{bmatrix} 3(3)+4(4) & 3(5)+4(4) & 3(5)+4(2) & 3(1)+4(6)\\ 1(3)+1(4) & 1(5)+1(4) & 1(5)+1(2) & 1(1)+1(6)\\ 6(3)+7(4) & 6(5)+7(4) & 6(5)+7(2) & 6(1)+7(6) \end{bmatrix} = \begin{bmatrix} 25 & 31 & 23 & 27\\ 7 & 9 & 7 & 7\\ 46 & 58 & 44 & 48 \end{bmatrix} $$

Permutation and Transformation Matricies

Matrix multiplication can be used to modify matrices with permutation and transformation matrices. Permutation and transformation matrices modify variables within a matrix using row operations. In the case of matrix multiplication, the modifying or operating matrix is multiplied on the left side of the system matrix. Permutation and transformation matrices do not change the dimension of the original matrix.

The Identity Matrix and Permutations

$$ A = \begin{bmatrix} 1 & 3 & 5\\ 2 & 2 & 4\\ 3 & 1 & 3 \end{bmatrix} $$

An identity matrix is equivalent to the value of 1 in scalar operations. The identity matrix is made up of ones along the diagonal on a matrix and zeroes in every other position. The following is a three-dimensional identity matrix:

$$ I = \begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & 0 & 1 \end{bmatrix} $$

When an identity matrix is multiplied into a matrix, the resulting product is the original matrix. A form of the identity matrix can be used to perform row swap operations. These matrices are known as permutation matrices. Consider the following permutation matrix that swaps the first and second rows in a three-dimensional system:

$$ P_{213} = \begin{bmatrix} 0 & 1 & 0\\ 1 & 0 & 0\\ 0 & 0 & 1 \end{bmatrix} $$

When this permutation matrix is multiplied into the matrix $ \textit{A} $, it will swap the first and second rows, while leaving the third row unchanged. Verify this for yourself using the following matrix product:

$$ P_{213} \times A = \begin{bmatrix} 0 & 1 & 0\\ 1 & 0 & 0\\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 3 & 5\\ 2 & 2 & 4\\ 3 & 1 & 3 \end{bmatrix} = \begin{bmatrix} 2 & 2 & 4\\ 1 & 3 & 5\\ 3 & 1 & 3 \end{bmatrix} $$

Five other permutation matrices exist for three-dimensional row swaps. Subscripts can be used to define the new row order after a row swap operation. For example, the following permutation matrix swaps the first, second, and third rows of a matrix:

$$ P_{312} = \begin{bmatrix} 0 & 0 & 1\\ 1 & 0 & 0\\ 0 & 1 & 0 \end{bmatrix} $$

Transformation Matrices

The following matrix $ \textit{M} $ is a two-dimensional system with a third dimension appended to allow for transformations. The first row can be considered the x-coordinates of a set of vectors, and the second row the y-components of those same vectors. This matrix will be used as an example of moving systems of points using transformation matrices.

$$ M = \begin{bmatrix} 2 & 3\\ 1 & 5\\ 1 & 1 \end{bmatrix} $$

A transformation matrix uses row operations to modify elements of a system. This operating matrix is a key concept in Gauss-Jordan elimination and reduced row echelon operations. Transformation matrices can also be used to scale rows of a matrix. Consider the following transformation matrix:

$$ E_1 = \begin{bmatrix} 1 & 0 & 3\\ 0 & 1 & 0\\ 0 & 0 & 1 \end{bmatrix} $$

The matrix $ E_1 $ moves the points of a system 3 units to the right. In other words, it transforms the x-components of the system. When multiplied with matrix $ \textit{M} $, the following transformed matrix is created:

$$ M_1 = E_1 \times M = \begin{bmatrix} 1 & 0 & 3\\ 0 & 1 & 0\\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 2 & 3\\ 1 & 5\\ 1 & 1 \end{bmatrix} = \begin{bmatrix} 5 & 6\\ 1 & 5\\ 1 & 1 \end{bmatrix} $$

A helpful way to consider the performed operation is by observing the dot products: the transformation matrix added 3 times the third row of $ \textit{M} $ to the first row.

$$ E_2 = \begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & -5\\ 0 & 0 & 1 \end{bmatrix} $$

This next transformation matrix moves the points of a two-dimensional system down by five units. Multiplying this into $ M_1 $ yields the following:

$$ M_2 = E_2 \times M_1 = \begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & -5\\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 5 & 6\\ 1 & 5\\ 1 & 1 \end{bmatrix} = \begin{bmatrix} 5 & 6\\ -4 & 0\\ 1 & 1 \end{bmatrix} $$

Transformation matrices can also scale rows. The next matrix scales the x-components by 2 and the y-components by $ \frac{1}{4} $:

$$ E_3 = \begin{bmatrix} 2 & 0 & 0\\ 0 & \frac{1}{4} & 0\\ 0 & 0 & 1 \end{bmatrix} $$

$$ M_3 = E_3 \times M_2 = \begin{bmatrix} 2 & 0 & 0\\ 0 & \frac{1}{4} & 0\\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 5 & 6\\ -4 & 0\\ 1 & 1 \end{bmatrix} = \begin{bmatrix} 10 & 12\\ -1 & 0\\ 1 & 1 \end{bmatrix} $$

As all of these transformation matrices were performed in order, they can be multiplied together in the same order to create a single transformation matrix that performs every operation simultaneously. This concept is used when finding inverses of a matrix through elimination matrices.

$$ E = E_3 \times E_2 \times E_1 = \begin{bmatrix} 2 & 0 & 0\\ 0 & \frac{1}{4} & 0\\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & -5\\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 0 & 3\\ 0 & 1 & 0\\ 0 & 0 & 1 \end{bmatrix} = \begin{bmatrix} 2 & 0 & 6\\ 0 & \frac{1}{4} & -\frac{5}{4}\\ 0 & 0 & 1 \end{bmatrix} $$

Using this transformation matrix, we can find the same matrix $ M_3 $ with a single operation:

$$ E \times M = \begin{bmatrix} 2 & 0 & 6\\ 0 & \frac{1}{4} & -\frac{5}{4}\\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 2 & 3\\ 1 & 5\\ 1 & 1 \end{bmatrix} = \begin{bmatrix} 10 & 12\\ -1 & 0\\ 1 & 1 \end{bmatrix} = M_3 $$

Gauss-Jordan Elimination and Reduced Row Echelon Form

Gauss-Jordan Elimination and Reduced row echelon matrices are useful in solving systems of equations. Consider the following system of equations.

\[x + 2y +2z = 6\]
\[3x + 5y - z = 2\]
\[2x + 2y + 5z = 9\]

To rewrite the system of equations in Matrix form, we can write a vector product or use an augmented matrix.

\[Ax=b\]

$$ \begin{bmatrix} 1 & 2 & 2\\ 3 & 5 & -1\\ 2 & 2 & 5 \end{bmatrix} \begin{bmatrix} x\\ y\\ z \end{bmatrix} = \begin{bmatrix} 6\\ -2\\ 9 \end{bmatrix} $$

$\textit{As an Augmented Matrix}$

$$ \begin{bmatrix} 1 & 2 & 2 & | & 6\\ 3 & 5 & -1 & | & 2\\ 2 & 2 & 5 & | & 9 \end{bmatrix} $$

To solve this system of equations using Gauss-Jordan Elimination, row operations are used to eliminate values below the pivot points or the variables along the diagonal of a matrix. The pivot points for the system of equations are the $\textit{1, 5, and 5}$. In this example, multiple row elimination operations will be performed simultaneously. This example will use the augmented matrix.

$$ A_{aug} = \begin{bmatrix} 1 & 2 & 2 & | & 6\\ 3 & 5 & -1 & | & 2\\ 2 & 2 & 5 & | & 9 \end{bmatrix} $$

$$ A_1=E_1 \times A_{aug} = \begin{bmatrix} 1 & 0 & 0\\ -3 & 1 & 0\\ -2 & 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 2 & 2 & | & 6\\ 3 & 5 & -1 & | & 2\\ 2 & 2 & 5 & | & 9 \end{bmatrix} = \begin{bmatrix} 1 & 2 & 2 & | & 6\\ 0 & -1 & -7 & | & -16\\ 0 & -2 & 1 & | & -3 \end{bmatrix} $$

$$ A_2=E_2\times A_1 = \begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & -2 & 1 \end{bmatrix} \begin{bmatrix} 1 & 2 & 2 & | & 6\\ 0 & -1 & -7 & | & -16\\ 0 & -2 & 1 & | & -3 \end{bmatrix} = \begin{bmatrix} 1 & 2 & 2 & | & 6\\ 0 & -1 & -7 & | & -16\\ 0 & 0 & 15 & | & 29 \end{bmatrix} $$

After the above row operations, the matrix is reduced to an upper right-hand triangular form. Back substitution can now be used to solve for the values of $\textit{x, y,}$ and $\textit{z}$. These values are associated with the first, second, and third columns of the augmented matrix. To solve with back-substitution, start with the row with the least number of non-zero values and move up solving for the missing variables.

\[15z=29\]
\[z = \frac{29}{15}\]
Using the second row and the solved $\textit{z}$ value
\[-y -7\bigg(\frac{29}{15}\bigg)=-16\]
\[y=16-\frac{203}{15}=\frac{37}{15}\]
Using the top row and the solved values of $\textit{y}$ and $\textit{z}$
\[x + 2 \bigg(\frac{37}{15}\bigg)+2\bigg(\frac{29}{15}\bigg)=6\]
\[x = 6 - \frac{132}{15}=-\frac{42}{15}=-\frac{14}{5}\]

These solved values give the solution to the system of equations. This solution method is satisfactory but the matrix can be further reduced. This reduction will result in an identity matrix on the left-hand side of the augmented matrix and the solution to the system on the right-hand side. The full reduction will continue from the previous matrix reduction step. From this point, the pivot points will be used to eliminate the remaining values in the upper right-hand triangle.

$$ A_3=E_3\times A_2 = \begin{bmatrix} 1 & 0 & -\frac{2}{15}\\ 0 & 1 & \frac{7}{15}\\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 2 & 2 & | & 6\\ 0 & -1 & -7 & | & -16\\ 0 & 0 & 15 & | & 29 \end{bmatrix} = \begin{bmatrix} 1 & 2 & 0 & | & \frac{32}{15}\\ 0 & -1 & 0 & | & -\frac{37}{15}\\ 0 & 0 & 15 & | & 29 \end{bmatrix} $$

$$ A_4=E_4\times A_3= \begin{bmatrix} 1 & 2 & 0\\ 0 & 1 & 0\\ 0 & 0 & 1 \end{bmatrix} \begin{bmatrix} 1 & 2 & 0 & | & \frac{32}{15}\\ 0 & -1 & 0 & | & -\frac{37}{15}\\ 0 & 0 & 15 & | & 29 \end{bmatrix} = \begin{bmatrix} 1 & 0 & 0 & | & -\frac{14}{5}\\ 0 & -1 & 0 & | & -\frac{37}{15}\\ 0 & 0 & 15 & | & 29 \end{bmatrix} $$

To finish solving the reduced row echelon form of matrix $\textit{A}$, the pivot points need to be scaled to positive 1.

$$ rref(A)=E_5\times A_4= \begin{bmatrix} 1 & 0 & 0\\ 0 & -1 & 0\\ 0 & 0 & \frac{1}{15} \end{bmatrix} \begin{bmatrix} 1 & 0 & 0 & | & -\frac{14}{5}\\ 0 & -1 & 0 & | & -\frac{37}{15}\\ 0 & 0 & 15 & | & 29 \end{bmatrix} = \begin{bmatrix} 1 & 0 & 0 & | & -\frac{14}{5}\\ 0 & 1 & 0 & | & \frac{37}{15}\\ 0 & 0 & 1 & | & \frac{29}{15} \end{bmatrix} $$

The values on the right-hand side match the solution derived using back substitution. The steps taken to fully reduce a matrix into this form can also be used to find a matrix $\textit{Inverse}$. An inverted matrix can be used to find an infinite number of solutions to a system based on a square matrix $\textit{A}$. This process will be explained in the following section.

Matrix Inversion

As matrix math does not allow for division, inverse matrices can be used as a method for solving systems of equations. Inverse matrices, when multiplied into an associated non-singular matrix, reduce that matrix into an identity. Any non-singular matrix has a unique inverse associated with it. Inverse matrices can be used to quickly solve systems of equations with different solution vectors. Consider the matrix $\textit{M}$ and its associated inverse below.

$$ M = \begin{bmatrix} 1 & 2\\ 2 & 3 \end{bmatrix} $$

$$ M^{-1} = \begin{bmatrix} -3 & 2\\ 2 & -1 \end{bmatrix} $$

These matrices can be multiplied on either side of each other and will always produce an identity matrix. We can use this inverse matrix to solve a system of equations based on the matrix $\textit{M}$ through matrix algebra.

\[Mx = b\]

$$ \begin{bmatrix} 1 & 2\\ 2 & 3 \end{bmatrix} \begin{bmatrix} x\\ y \end{bmatrix} = \begin{bmatrix} 4\\ 6 \end{bmatrix} $$

\[M^{-1}Mx = M^{-1}b\]
\[Ix = M^{-1}b\]

$$ \begin{bmatrix} x\\ y \end{bmatrix} = \begin{bmatrix} -3 & 2\\ 2 & -1 \end{bmatrix} \begin{bmatrix} 4\\ 6 \end{bmatrix} = \begin{bmatrix} 0\\ 2 \end{bmatrix} $$

This solution vector gives the values of $\textit{x}$ and $\textit{y}$ to the system based on the value of $\textit{b}$.

Finding a Matrix Inverse

Inverse matrices are a powerful tool for system analysis. Solving for an inverse can become complicated and arduous. To start, consider a symbolic $2 \times 2$ matrix.

$$ S = \begin{bmatrix} a & b\\ c & d \end{bmatrix} $$

To solve the inverse of this matrix, we can consider the steps it takes to reduce the matrix using the reduced row echelon form. To start, augment the matrix $\textit{S}$ with an identity matrix.

$$ S_{\text{Aug}} = \begin{bmatrix} a & b & | & 1 & 0\\ c & d & | & 0 & 1 \end{bmatrix} $$

Follow the steps for row reduction:

$$ E_1 = \begin{bmatrix} 1 & 0\\ -\frac{c}{a} & 1 \end{bmatrix} $$

$$ S_1 = E_1 \times S_{\text{Aug}} = \begin{bmatrix} a & b & | & 1 & 0\\ 0 & \frac{da - bc}{a} & | & -\frac{c}{a} & 1 \end{bmatrix} $$

$$ E_2 = \begin{bmatrix} 1 & -\frac{ba}{da - bc}\\ 0 & 1 \end{bmatrix} $$

$$ S_2 = E_2 \times S_1 = \begin{bmatrix} a & 0 & | & \frac{da}{da - bc} & -\frac{ba}{da - bc}\\ 0 & \frac{da - bc}{a} & | & -\frac{c}{a} & 1 \end{bmatrix} $$

$$ E_3 = \begin{bmatrix} \frac{1}{a} & 0\\ 0 & \frac{a}{da - bc} \end{bmatrix} $$

$$ S^{-1} = E_3 \times S_2 = \begin{bmatrix} 1 & 0 & | & \frac{d}{da - bc} & -\frac{b}{da - bc}\\ 0 & 1 & | & -\frac{c}{da - bc} & \frac{a}{da - bc} \end{bmatrix} $$

The result of taking the reduced row echelon form of the augmented matrix is an identity matrix on the left-hand side and the inverse of $\textit{S}$ on the right. This symbolic form of an inverse $2\times 2$ matrix is applicable to any non-singular matrix. This can also be written in a form that relates the inverse of the matrix to its determinant:

$$ S^{-1} = \frac{1}{\det(A)} \begin{bmatrix} d & -b\\ -c & a \end{bmatrix} $$

Due to complexity, the inverse of a square matrix with higher dimensionality is not reasonable to derive or memorize. As such, an example of finding the inverse of a $3 \times 3$ matrix will be given. Recall the $3 \times 3$ matrix used for Gauss-Jordan elimination and its elimination matrices.

$$ A = \begin{bmatrix} 1 & 2 & 2\\ 3 & 5 & -1\\ 2 & 2 & 5 \end{bmatrix} $$

$$ E_1 = \begin{bmatrix} 1 & 0 & 0\\ -3 & 1 & 0\\ -2 & 0 & 1 \end{bmatrix} $$

$$ E_2 = \begin{bmatrix} 1 & 0 & 0\\ 0 & 1 & 0\\ 0 & -2 & 1 \end{bmatrix} $$

$$ E_3 = \begin{bmatrix} 1 & 0 & -\frac{2}{15}\\ 0 & 1 & \frac{7}{15}\\ 0 & 0 & 1 \end{bmatrix} $$

$$ E_4 = \begin{bmatrix} 1 & 2 & 0\\ 0 & 1 & 0\\ 0 & 0 & 1 \end{bmatrix} $$

$$ E_5 = \begin{bmatrix} 1 & 0 & 0\\ 0 & -1 & 0\\ 0 & 0 & \frac{1}{15} \end{bmatrix} $$

The steps taken in finding the reduced row echelon form of $\textit{A}$ are the same as those used in finding an inverse matrix. While there are multiple ways to solve for a matrix inverse such as LU Decomposition, those methods will not be discussed here. To find the inverse of $\textit{A}$ from the given elimination matrices, multiply the elimination matrices in the same order that they were derived.

$$ A^{-1} = E_5 \times E_4 \times E_3 \times E_2 \times E_1 = \begin{bmatrix} -\frac{9}{5} & \frac{2}{5} & \frac{4}{5}\\ \frac{17}{15} & -\frac{1}{15} & -\frac{7}{15}\\ \frac{4}{15} & -\frac{2}{15} & \frac{1}{15} \end{bmatrix} $$

To check if a matrix is an inverse, multiply it into the associated matrix that the inverse was derived from. This can allow you to ensure that it is a valid inverse. This inverse matrix of $\textit{A}$ can also be used to find the system solution.

\[Ax = b\]
\[x = A^{-1}b\]

$$ \begin{bmatrix} x\\ y\\ z \end{bmatrix} = \begin{bmatrix} -\frac{9}{5} & \frac{2}{5} & \frac{4}{5}\\ \frac{17}{15} & -\frac{1}{15} & -\frac{7}{15}\\ \frac{4}{15} & -\frac{2}{15} & \frac{1}{15} \end{bmatrix} \begin{bmatrix} 6\\ 2\\ 9 \end{bmatrix} = \begin{bmatrix} -\frac{14}{5}\\ \frac{37}{15}\\ \frac{29}{15} \end{bmatrix} $$

Matrix Multiplication
Matrix Multiplication Examples
Permutation and Transformation Matricies
The Identity Matrix and Permutations
Transformation Matricies
Gauss-Jordan Elimination and Reduced Row Echelon Form
Matrix Inversion
Finding a Matrix Inverse