Skip to main content

MATH 345: Linear Algebra and Optimization

Sebastien Roch with Jacob Denson and Kaitlyn Phillipson

Section 4.1 Orthogonality and orthogonal bases

Subsection Basic definitions

Definition 4.1.1.

Recall the definition of the dot product of two vectors, introduced in Definition 1.1.10, and the length of a vector as introduced in Definition 1.1.3. The distance between two points \(\mathbf{x} = (x_1,\dots,x_n)\) and \(\mathbf{y} = (y_1,\cdots,y_n)\) is the length of the vector \(\mathbf{x} - \mathbf{y}\text{,}\) i.e.,
\begin{equation*} d(\mathbf{x},\mathbf{y}) = \| \mathbf{x} - \mathbf{y} \| = \sqrt{ (x_1 - y_1)^2 + \cdots + (x_n - y_n)^2} \text{.} \end{equation*}
If \(\mathbf{x}\) is any non-zero vector, then the vector \(\mathbf{u} = \mathbf{x} / \| \mathbf{x} \|\) is the unit vector in the direction of \(\mathbf{x}\) (a unit vector is a vector of length one). Two vectors \(\mathbf{x}\) and \(\mathbf{y}\) are orthogonal or perpendicular if \(\mathbf{x} \cdot \mathbf{y} = 0\text{.}\)
🔗
🔗

Activity 4.1.1. Cosine similarity revisited.

Let \(\mathbf{u}\) and \(\mathbf{v}\) be nonzero vectors in \(\R^n\text{.}\) The cosine similarity from Unit 1 was
\begin{equation*} \frac{\mathbf{u}\cdot\mathbf{v}}{\|\mathbf{u}\|\|\mathbf{v}\|}. \end{equation*}
🔗
  1. Use the Cauchy-Schwarz inequality to explain why this number is between \(-1\) and \(1\text{.}\)
    🔗
    🔗
  2. What condition makes the cosine similarity equal to \(0\text{?}\)
    🔗
    🔗
  3. In Unit 1, this number measured cosine similarity. What does a value of \(0\) say geometrically?
    🔗
    🔗
Solution.
By Cauchy-Schwarz, \(|\mathbf{u}\cdot\mathbf{v}|\leq \|\mathbf{u}\|\|\mathbf{v}\|\text{.}\) Since both norms are nonzero, division gives \(-1\leq(\mathbf{u}\cdot\mathbf{v})/(\|\mathbf{u}\|\|\mathbf{v}\|)\leq 1\text{.}\) The value is \(0\) exactly when \(\mathbf{u}\cdot\mathbf{v}=0\text{,}\) which means the vectors are orthogonal.
🔗
🔗
🔗

Remark 4.1.4.

The triangle inequality for vectors implies the triangle inequality for the distance function, i.e., that
\begin{equation*} d(\mathbf{x}, \mathbf{z}) \leq d(\mathbf{x}, \mathbf{y}) + d(\mathbf{y}, \mathbf{z})\text{.} \end{equation*}
Indeed, we calculate that
\begin{align*} d(\mathbf{x}, \mathbf{z}) \amp = \| \mathbf{x} - \mathbf{z} \|\\ \amp = \| \mathbf{x} - \mathbf{y} + \mathbf{y} - \mathbf{z} \|\\ \amp \leq \| \mathbf{x} - \mathbf{y} \| + \| \mathbf{y} - \mathbf{z} \|\text{.} \end{align*}
🔗
🔗

Activity 4.1.2.

Show that \(\| \mathbf{x} + \mathbf{y} \|^2 = \| \mathbf{x} \|^2 + 2 (\mathbf{x} \cdot \mathbf{y}) + \| \mathbf{y} \|^2\) for any \(\mathbf{x}, \mathbf{y} \in \R^n\text{.}\)
🔗
Solution.
We calculate using the properties in Theorem 4.1.5 that
\begin{align*} \| \mathbf{x} + \mathbf{y} \|^2 \amp = ( \mathbf{x} + \mathbf{y} ) \cdot ( \mathbf{x} + \mathbf{y} )\\ \amp = \mathbf{x} \cdot \mathbf{x} + \mathbf{y} \cdot \mathbf{x} + \mathbf{x} \cdot \mathbf{y} + \mathbf{y} \cdot \mathbf{y}\\ \amp = \| \mathbf{x} \|^2 + 2 \mathbf{x} \cdot \mathbf{y} + \| \mathbf{y} \|^2\text{.} \end{align*}
🔗
🔗
🔗

Activity 4.1.3.

Suppose that \(\R^n = \spans \{ \mathbf{f}_1,\dots,\mathbf{f}_k \}\) for some vectors \(\mathbf{f}_i\text{.}\) If \(\mathbf{x} \in \R^n\text{,}\) and \(\mathbf{x} \cdot \mathbf{f}_i = 0\) for each \(i\text{,}\) then show \(\mathbf{x} = \mathbf{0}\text{.}\)
🔗
Solution.
Since \(\R^n\) is spanned by the vectors \(\{ \mathbf{f}_1,\dots,\mathbf{f}_k \}\text{,}\) we can write \(\mathbf{x} = t_1 \mathbf{f}_1 + \cdots + t_k \mathbf{f}_k\) for some scalars \(t_1,\dots,t_k \in \R\text{.}\) But then
\begin{align*} \| \mathbf{x} \|^2 \amp = \mathbf{x} \cdot \mathbf{x}\\ \amp = \mathbf{x} \cdot ( t_1 \mathbf{f}_1 + \cdots + t_k \mathbf{f}_k )\\ \amp = t_1 (\mathbf{x} \cdot \mathbf{f}_1) + \cdots + t_k (\mathbf{x} \cdot \mathbf{f}_k)\\ \amp = t_1 (0 ) + \cdots + t_k (0)\\ \amp = 0\text{.} \end{align*}
Thus \(\| \mathbf{x} \| = 0\text{.}\) But this can only occur if \(\mathbf{x} = \mathbf{0}\) by Theorem 4.1.5.
🔗
🔗
🔗
🔗

Subsection Orthogonal sets and the expansion theorem

Definition 4.1.6. Orthogonal and orthonormal sets.

A set of vectors \(\{ \mathbf{x}_1, \dots, \mathbf{x}_k \}\) is called an orthogonal set if the set does not contain the zero vector, and \(\mathbf{x}_i \cdot \mathbf{x}_j = 0\) for \(i \neq j\text{.}\) A set \(\{ \mathbf{x}_1, \dots, \mathbf{x}_k \}\) is called orthonormal if it is orthogonal and if in addition, \(\| \mathbf{x}_i \| = 1\) for each \(i\text{.}\)
🔗
🔗
Note that \(\{ \mathbf{x} \}\) is an orthogonal set if \(\mathbf{x} \neq \mathbf{0}\text{.}\) The standard basis \(\{ \mathbf{e}_1, \cdots, \mathbf{e}_n \}\) (see Definition 1.3.8) is an important orthonormal set in \(\R^n\text{.}\)
🔗
🔗

Subsection Orthogonal sets and bases

Activity 4.1.4.

Consider the following vectors in \(\R^4\text{,}\) written as column vectors:
\begin{equation*} \mathbf{f}_1 = \begin{bmatrix} 1 \\ 1 \\ 1 \\ -1 \end{bmatrix} \quad \mathbf{f}_2 = \begin{bmatrix} 1 \\ 0 \\ 1 \\ 2 \end{bmatrix} \quad \mathbf{f}_3 = \begin{bmatrix} -1 \\ 0 \\ 1 \\ 0 \end{bmatrix} \quad \mathbf{f}_4 = \begin{bmatrix} -1 \\ 3 \\ -1 \\ 1 \end{bmatrix}\text{.} \end{equation*}
Verify that these vectors form an orthogonal set in \(\R^4\text{.}\) Then find the orthonormal set obtained by ‘normalizing’ these vectors, i.e., multiplying each vector by an appropriate constant to obtain a unit vector.
🔗
Solution.
To verify that \(\{\mathbf{f}_1, \mathbf{f}_2, \mathbf{f}_3, \mathbf{f}_4\}\) form an orthogonal set, we need to check that \(\mathbf{f}_i \cdot \mathbf{f}_j = 0\) for all \(i \neq j\text{.}\) We calculate that
\begin{align*} \mathbf{f}_1 \cdot \mathbf{f}_2 \amp = 1 \cdot 1 + 1 \cdot 0 + 1 \cdot 1 + (-1) \cdot 2 = 1 + 0 + 1 - 2 = 0\\ \mathbf{f}_1 \cdot \mathbf{f}_3 \amp = 1 \cdot (-1) + 1 \cdot 0 + 1 \cdot 1 + (-1) \cdot 0 = -1 + 0 + 1 + 0 = 0\\ \mathbf{f}_1 \cdot \mathbf{f}_4 \amp = 1 \cdot (-1) + 1 \cdot 3 + 1 \cdot (-1) + (-1) \cdot 1 = -1 + 3 - 1 - 1 = 0\\ \mathbf{f}_2 \cdot \mathbf{f}_3 \amp = 1 \cdot (-1) + 0 \cdot 0 + 1 \cdot 1 + 2 \cdot 0 = -1 + 0 + 1 + 0 = 0\\ \mathbf{f}_2 \cdot \mathbf{f}_4 \amp = 1 \cdot (-1) + 0 \cdot 3 + 1 \cdot (-1) + 2 \cdot 1 = -1 + 0 - 1 + 2 = 0\\ \mathbf{f}_3 \cdot \mathbf{f}_4 \amp = (-1) \cdot (-1) + 0 \cdot 3 + 1 \cdot (-1) + 0 \cdot 1 = 1 + 0 - 1 + 0 = 0\text{.} \end{align*}
Since all these dot products are zero, the set is orthogonal.
🔗
To normalize these vectors, we need to calculate their lengths and divide each vector by its length. We find that
\begin{align*} \|\mathbf{f}_1\| \amp = \sqrt{1^2 + 1^2 + 1^2 + (-1)^2} = \sqrt{4} = 2\\ \|\mathbf{f}_2\| \amp = \sqrt{1^2 + 0^2 + 1^2 + 2^2} = \sqrt{6}\\ \|\mathbf{f}_3\| \amp = \sqrt{(-1)^2 + 0^2 + 1^2 + 0^2} = \sqrt{2}\\ \|\mathbf{f}_4\| \amp = \sqrt{(-1)^2 + 3^2 + (-1)^2 + 1^2} = \sqrt{12} = 2\sqrt{3}\text{.} \end{align*}
Therefore the normalized vectors are given by:
\begin{align*} \frac{1}{\|\mathbf{f}_1\|} \mathbf{f}_1 \amp = \frac{1}{2}\mathbf{f}_1 = \begin{bmatrix} \frac{1}{2} \\ \frac{1}{2} \\ \frac{1}{2} \\ -\frac{1}{2} \end{bmatrix}\\ \frac{1}{\|\mathbf{f}_2\|} \mathbf{f}_2 \amp = \frac{1}{\sqrt{6}}\mathbf{f}_2 = \begin{bmatrix} \frac{1}{\sqrt{6}} \\ 0 \\ \frac{1}{\sqrt{6}} \\ \frac{2}{\sqrt{6}} \end{bmatrix}\\ \frac{1}{\|\mathbf{f}_3\|} \mathbf{f}_3 \amp = \frac{1}{\sqrt{2}}\mathbf{f}_3 = \begin{bmatrix} -\frac{1}{\sqrt{2}} \\ 0 \\ \frac{1}{\sqrt{2}} \\ 0 \end{bmatrix}\\ \frac{1}{\|\mathbf{f}_4\|} \mathbf{f}_4 \amp = \frac{1}{2\sqrt{3}}\mathbf{f}_4 = \begin{bmatrix} -\frac{1}{2\sqrt{3}} \\ \frac{3}{2\sqrt{3}} \\ -\frac{1}{2\sqrt{3}} \\ \frac{1}{2\sqrt{3}} \end{bmatrix}\text{.} \end{align*}
🔗
🔗
🔗

Why is this true?.

Let \(\{\mathbf{f}_1, \mathbf{f}_2, \ldots, \mathbf{f}_k\}\) be an orthogonal set in \(\R^n\text{.}\) To show the set is linearly independent (recalling Definition 2.4.1), we must show the only solution to the linear equation
\begin{equation*} t_1\mathbf{f}_1 + t_2\mathbf{f}_2 + \cdots + t_k\mathbf{f}_k = \mathbf{0} \end{equation*}
occurs when \(t_1 = \cdots = t_k = 0\text{.}\)
🔗
If we take the dot product of both sides of the equation with \(\mathbf{f}_1\text{,}\) we find that
\begin{equation*} \mathbf{f}_1 \cdot (t_1\mathbf{f}_1 + t_2\mathbf{f}_2 + \cdots + t_k\mathbf{f}_k) = \mathbf{f}_1 \cdot \mathbf{0} \end{equation*}
and therefore that
\begin{equation*} t_1(\mathbf{f}_1 \cdot \mathbf{f}_1) + t_2(\mathbf{f}_1 \cdot \mathbf{f}_2) + \cdots + t_k(\mathbf{f}_1 \cdot \mathbf{f}_k) = 0 \end{equation*}
Since the set of vectors is orthogonal, \(\mathbf{f}_1 \cdot \mathbf{f}_j = 0\) for all \(j \neq 1\text{,}\) and \(\mathbf{f}_1 \cdot \mathbf{f}_1 = \| \mathbf{f}_1 \|^2\text{,}\) so we conclude that
\begin{equation*} t_1\|\mathbf{f}_1\|^2 = 0. \end{equation*}
Since \(\mathbf{f}_1 \neq \mathbf{0}\) (by definition, an orthogonal set cannot contain the zero vector), \(\|\mathbf{f}_1\|^2 > 0\text{,}\) and so it must be true that \(t_1 = 0\text{.}\) But now repeating this argument with \(1\) replaced by any index \(i\text{,}\) we conclude that \(t_i = 0\text{.}\) Therefore the set \(\{ \mathbf{f}_1, \dots, \mathbf{f}_k \}\) is linearly independent.
🔗
🔗

Why is this true?.

Since \(\{\mathbf{f}_1, \mathbf{f}_2, \ldots, \mathbf{f}_m\}\) spans \(U\text{,}\) any \(\mathbf{v} \in U\) can be written uniquely as
\begin{equation*} \mathbf{v} = t_1\mathbf{f}_1 + t_2\mathbf{f}_2 + \cdots + t_m\mathbf{f}_m \end{equation*}
for some \(t_1,\dots,t_m\text{.}\) To find coefficient \(t_1\text{,}\) take the dot product of both sides with \(\mathbf{f}_1\text{,}\) obtaining that
\begin{align*} \mathbf{v} \cdot \mathbf{f}_1 \amp = (t_1\mathbf{f}_1 + t_2\mathbf{f}_2 + \cdots + t_m\mathbf{f}_m) \cdot \mathbf{f}_1\\ \amp = t_1(\mathbf{f}_1 \cdot \mathbf{f}_1) + t_2(\mathbf{f}_2 \cdot \mathbf{f}_1) + \cdots + t_m(\mathbf{f}_m \cdot \mathbf{f}_1)\text{.} \end{align*}
Since the basis is orthogonal, \(\mathbf{f}_i \cdot \mathbf{f}_j = 0\) for \(i \neq j\text{,}\) and so the equation simplifies to
\begin{equation*} \mathbf{v} \cdot \mathbf{f}_1 = t_1\|\mathbf{f}_1\|^2\text{.} \end{equation*}
Rearranging, we conclude that
\begin{equation*} t_1 = \frac{\mathbf{v} \cdot \mathbf{f}_1}{\|\mathbf{f}_1\|^2} \end{equation*}
A similar argument shows that
\begin{equation*} t_i = \frac{\mathbf{v} \cdot \mathbf{f}_i}{\|\mathbf{f}_i\|^2} \end{equation*}
for each \(i\text{,}\) concluding the proof.
🔗
🔗

Definition 4.1.10. The Fourier expansion in an orthogonal basis.

Given an orthogonal basis \(\{ \mathbf{f}_1, \dots, \mathbf{f}_m \}\) for some subspace \(V\) of \(\R^n\text{,}\) the expansion of \(\mathbf{v} \in V\) as a linear combination of this basis is called the Fourier expansion of the vector \(\mathbf{v}\text{,}\) and the coefficients
\begin{equation*} t_i = \frac{\mathbf{v} \cdot \mathbf{f}_i}{\|\mathbf{f}_i\|^2} \end{equation*}
are called the Fourier coefficients of \(\mathbf{v}\text{.}\)
🔗
🔗

Activity 4.1.5.

Let \(\mathbf{v} = (a, b, c, d)\) be a vector in \(\R^4\text{.}\) Find the Fourier expansion of \(\mathbf{v}\) as a linear combination of the orthogonal basis \(\{\mathbf{f}_1, \mathbf{f}_2, \mathbf{f}_3, \mathbf{f}_4\}\) given in Activity 4.1.4.
🔗
Solution.
We need to find the Fourier coefficients
\begin{equation*} t_i = \frac{\mathbf{v} \cdot \mathbf{f}_i}{\|\mathbf{f}_i\|^2} \end{equation*}
for \(i = 1, 2, 3, 4\text{.}\) In Activity 4.1.4 we already computed the lengths of the elements of the basis, namely, that
\begin{equation*} \| \mathbf{f}_1 \| = 2, \quad \| \mathbf{f}_2 \| = \sqrt{6}, \quad \| \mathbf{f}_3 \| = \sqrt{2} \quad\text{and}\quad \| \mathbf{f}_4 \| = 2 \sqrt{3}\text{.} \end{equation*}
We now calculate the dot products:
\begin{align*} \mathbf{v} \cdot \mathbf{f}_1 \amp = a \cdot 1 + b \cdot 1 + c \cdot 1 + d \cdot (-1) = a + b + c - d\\ \mathbf{v} \cdot \mathbf{f}_2 \amp = a \cdot 1 + b \cdot 0 + c \cdot 1 + d \cdot 2 = a + c + 2d\\ \mathbf{v} \cdot \mathbf{f}_3 \amp = a \cdot (-1) + b \cdot 0 + c \cdot 1 + d \cdot 0 = -a + c\\ \mathbf{v} \cdot \mathbf{f}_4 \amp = a \cdot (-1) + b \cdot 3 + c \cdot (-1) + d \cdot 1 = -a + 3b - c + d \end{align*}
Therefore, the Fourier coefficients are:
\begin{align*} t_1 \amp = \frac{\mathbf{v} \cdot \mathbf{f}_1}{\|\mathbf{f}_1\|^2} = \frac{a + b + c - d}{4}\\ t_2 \amp = \frac{\mathbf{v} \cdot \mathbf{f}_2}{\|\mathbf{f}_2\|^2} = \frac{a + c + 2d}{6}\\ t_3 \amp = \frac{\mathbf{v} \cdot \mathbf{f}_3}{\|\mathbf{f}_3\|^2} = \frac{-a + c}{2}\\ t_4 \amp = \frac{\mathbf{v} \cdot \mathbf{f}_4}{\|\mathbf{f}_4\|^2} = \frac{-a + 3b - c + d}{12} \end{align*}
The Fourier expansion of \(\mathbf{v}\) in the basis \(\{ \mathbf{f}_1, \dots, \mathbf{f}_4 \}\) is therefore
\begin{align*} \mathbf{v} \amp = \left(\frac{a + b + c - d}{4} \right)\mathbf{f}_1 + \left(\frac{a + c + 2d}{6}\right)\mathbf{f}_2\\ \amp \quad \quad + \left(\frac{-a + c}{2}\right)\mathbf{f}_3 + \left(\frac{-a + 3b - c + d}{12}\right) \mathbf{f}_4\text{.} \end{align*}
🔗
🔗
🔗
🔗
Dot products and orthogonality now become tools for finding closest vectors.
🔗
🔗
PrevTopNext