Inner product spaces

Section 7.2 Inner product spaces

Unit 4 used the dot product on \(\mathbb R^n\) to define length, distance, orthogonality, projection, and least squares. An inner product space keeps exactly the rules needed for those ideas. This section introduces the abstract definition and keeps three examples in view: \(\mathbb R^n\text{,}\) polynomial spaces, and matrix spaces.

🔗

First, we define inner products.

🔗

Definition 7.2.1. Inner Products.

Let \(V\) be a vector space. An inner product on \(V\) is a function \(\langle \; , \; \rangle\) that assigns to each ordered pair of vectors \(\mathbf{v}\text{,}\) \(\mathbf{w}\) in \(V\) a real number \(\langle \mathbf{v}, \mathbf{w}\rangle\) satisfying the following properties:

\(\langle \mathbf{v}, \mathbf{w}\rangle=\langle \mathbf{w}, \mathbf{v}\rangle\) for all \(\mathbf{v}, \mathbf{w}\) in \(V\text{.}\)

🔗
\(\langle \mathbf{v}+\mathbf{w}, \mathbf{u}\rangle=\langle \mathbf{v}, \mathbf{u}\rangle + \langle \mathbf{w}, \mathbf{u}\rangle\) for all \(\mathbf{u}, \mathbf{v}, \mathbf{w}\) in \(V\text{.}\)

🔗
\(\langle r\mathbf{v}, \mathbf{w}\rangle=r \langle \mathbf{v}, \mathbf{w}\rangle\) for all \(\mathbf{v}\) and \(\mathbf{w}\) in \(V\) and \(r\) in \(\mathbb{R}\text{.}\)

🔗
\(\langle \mathbf{v}, \mathbf{v}\rangle > 0\) for all \(\mathbf{v}\neq\mathbf{0}\) in \(V\text{.}\)

🔗

🔗

The vector space \(V\) together with \(\langle \; , \; \rangle\) is called an inner product space.

🔗

Theorem 7.2.2.

For an inner product \(\langle \; , \; \rangle\) on a vector space \(V\text{,}\) the following additional properties automatically hold:

\(\langle\mathbf{u}, \mathbf{v}+\mathbf{w}\rangle=\langle\mathbf{u}, \mathbf{v}\rangle+\langle\mathbf{u}, \mathbf{w}\rangle\) for any \(\mathbf{u}, \mathbf{v}\) in \(V\text{.}\)

🔗
\(\langle\mathbf{u}, r\mathbf{v}\rangle=r\langle\mathbf{u}, \mathbf{v}\rangle\) for any \(\mathbf{u}, \mathbf{v}\) in \(V\text{,}\) and \(r\) in \(\mathbb{R}\text{.}\)

🔗
\(\displaystyle \langle \mathbf{v}, \mathbf{0}\rangle =0=\langle \mathbf{0},\mathbf{v}\rangle\)

🔗
\(\langle \mathbf{v}, \mathbf{v}\rangle=0\) if and only if \(\mathbf{v}=\mathbf{0}\text{.}\)

🔗

🔗

The dot product on \(\mathbb{R}^n\) is an inner product. We will often call the dot product the standard inner product.

🔗

Activity 7.2.1.

Let

\begin{equation*} \mathbf{u}=\begin{bmatrix} u_1 \\ u_2\end{bmatrix} \quad\text{and}\quad \mathbf{v}=\begin{bmatrix} v_1 \\v_2 \end{bmatrix} \end{equation*}

be vectors in \(\mathbb{R}^2\text{,}\) and define

\begin{equation*} \langle \mathbf{u},\mathbf{v}\rangle=u_1v_1-u_2v_1-u_1v_2+3u_2v_2 \end{equation*}

Show this definition satisfies the 4th property of Theorem 7.2.2, i.e., that \(\langle\mathbf{u},\mathbf{u}\rangle> 0\) for \(\mathbf{u}\neq \mathbf{0}\text{.}\)

🔗

Solution.

We calculate that

\begin{align*} \langle \mathbf{u}, \mathbf{u} \rangle \amp = u_1 u_1 - u_2 u_1 - u_1 u_2 + 3u_2 u_2\\ \amp = u_1^2 - 2u_1 u_2 + 3u_2^2\\ \amp = u_1^2 - 2u_1 u_2 + u_2^2 + 2u_2^2\\ \amp = (u_1 - u_2)^2 + 2u_2^2 \end{align*}

This quantity can only be zero if \(u_1 - u_2 = 0\) and \(u_2 = 0\text{.}\) But this is only possible if \(u_1 = 0\) and \(u_2 = 0\text{,}\) i.e., \(\mathbf{u} = \mathbf{0}\text{.}\)

🔗

Theorem 7.2.3.

Let \(E=\{\mathbf{e}_1, \mathbf{e}_2, \ldots, \mathbf{e}_n\}\) be the standard basis for \(\mathbb{R}^n\text{,}\) and let \(\langle \; , \; \rangle\) be any inner product on \(\mathbb{R}^n\text{.}\) Let \(c_{ij}=\langle\mathbf{e}_i, \mathbf{e}_j\rangle\) and consider the matrix \(C=[c_{ij}]\text{.}\) Then

\(C\) is a symmetric matrix.

🔗
\(\langle \mathbf{u},\mathbf{v}\rangle= \mathbf{u}^TC\mathbf{v}\) for all \(\mathbf{u},\mathbf{v}\) in \(\mathbb{R}^n\text{.}\)

🔗

🔗

Definition 7.2.4.

Given an inner product \(\langle \; , \; \rangle\) on \(\R^n\text{,}\) the symmetric matrix \(C\) analyzed in Theorem 7.2.3 is called the Gram matrix associated with the inner product.

🔗

Remark 7.2.5.

If a symmetric \(n \times n\) matrix \(C\) satisfies \(\mathbf{x}^T C \mathbf{x} > 0\) for all \(\mathbf{x} \neq \mathbf{0}\text{,}\) then \(C\) defines an inner product on \(\R^n\text{.}\) Moreover, these are the only symmetric matrices that define inner products. They are called positive definite matrices.

🔗

Definition 7.2.6. Positive Definite Matrices.

An \(n \times n\) matrix \(A\) is positive definite if it is symmetric, and if \(\mathbf{x}^T A \mathbf{x} > 0\) for all \(\mathbf{x} \neq 0\text{.}\)

🔗

Activity 7.2.2.

Consider the inner product on \(\mathbb{R}^2\) defined by

\begin{equation*} \langle\mathbf{u},\mathbf{v}\rangle=u_1v_1-u_2v_1-u_1v_2+3u_2v_2\text{.} \end{equation*}

Find \(C\) such that \(\langle \mathbf{u},\mathbf{v}\rangle=\mathbf{u}^TC\mathbf{v}\text{.}\)

🔗

Solution.

We employ Theorem 7.2.3. We compute that if \(C = [c_{ij} ]\text{,}\) then

\begin{align*} c_{11} \amp = \langle \mathbf{e}_1, \mathbf{e}_1 \rangle = (1)(1) - (0)(1) - (1)(0) + 3(0)(0) = 1\\ c_{12} \amp = \langle \mathbf{e}_1, \mathbf{e}_2 \rangle = (1)(0) - (0)(0) - (1)(1) + 3(0)(1) = -1\\ c_{21} \amp = \langle \mathbf{e}_2, \mathbf{e}_1 \rangle = (0)(1) - (1)(1) - (0)(0) + 3(1)(0) = -1\\ c_{22} \amp = \langle \mathbf{e}_2, \mathbf{e}_2 \rangle = (0)(0) - (1)(0) - (0)(1) + 3(1)(1) = 3 \end{align*}

🔗

Therefore,

\begin{equation*} C = \begin{bmatrix} 1 \amp -1 \\ -1 \amp 3 \end{bmatrix}\text{.} \end{equation*}

🔗

Definition 7.2.7. The Norm of a Vector.

In an inner product space, the norm of a vector \(\mathbf{v}\) is

\begin{equation*} \| \mathbf{v} \| = \sqrt{\langle\mathbf{v}, \mathbf{v}\rangle}. \end{equation*}

The distance between two vectors \(\mathbf{u}\) and \(\mathbf{v}\) in an inner product space is

\begin{equation*} d(\mathbf{u}, \mathbf{v}) = \| \mathbf{u}-\mathbf{v} \|. \end{equation*}

🔗

Definition 7.2.8. Unit Vectors.

A vector \(\mathbf{v}\) in an inner product space \(V\) is called a unit vector if \(\|\mathbf{v}\| = 1\text{.}\) This is equivalent to the equation \(\langle \mathbf{v}, \mathbf{v}\rangle =1\text{.}\) The set of all unit vectors in \(V\) is called the unit sphere in \(V\text{.}\)

🔗

Theorem 7.2.9.

If \(\mathbf{v}\) is a nonzero vector of an inner product space, then

\begin{equation*} \mathbf{u} = \frac{1}{\|\mathbf{v} \|}\mathbf{v} \end{equation*}

is the unique unit vector that is a positive multiple of \(\mathbf{v}\text{.}\) We often say that \(\mathbf{u}\) is the unit vector ‘in the same direction as \(\mathbf{v}\)’.

🔗

Definition 7.2.10. Orthogonal Vectors.

Two vectors \(\mathbf{u}\) and \(\mathbf{v}\) in an inner product space are orthogonal if

\begin{equation*} \langle \mathbf{u}, \mathbf{v}\rangle = 0\text{.} \end{equation*}

🔗

The next examples show that inner products are not limited to ordinary coordinate vectors. The objects can be polynomials or matrices, as long as the rule for \(\langle\cdot,\cdot\rangle\) satisfies the inner-product properties.

🔗

Subsection Spaces of polynomials

Activity 7.2.3.

Let \(V = \mathbf{P}_1\) be the vector space of all polynomials of degree at most \(1\text{.}\) Define

\begin{equation*} \langle f, g\rangle = \int_{0}^{1} f(x)g(x)\, dx\text{.} \end{equation*}

Does this definition define an inner product on \(V\text{?}\)

🔗

Solution.

Let

\begin{equation*} f(x) = a_0 + a_1 x \quad g(x) = b_0 + b_1 x \quad h(x) = c_0 + c_1 x \end{equation*}

be arbitrary polynomials in \(\mathbf{P}_1\text{.}\)

🔗

We begin by checking that property (a) of the definition of an inner product holds, i.e., that \(\langle f, g \rangle = \langle g, f \rangle\text{.}\) Indeed,

\begin{equation*} \langle f, g \rangle = \int_0^1 f(x) g(x)\; dx = \int_0^1 g(x) f(x)\; dx = \langle g, f \rangle\text{.} \end{equation*}

To verify that property (b) holds, we check that \(\langle f + g, h \rangle = \langle f, h \rangle + \langle g, h \rangle\text{.}\) We calculate that

\begin{align*} \langle f+g, h\rangle \amp = \int_0^1 (f(x)+g(x))h(x)\, dx\\ \amp = \int_0^1 f(x)h(x)\, dx + \int_0^1 g(x)h(x)\, dx\\ \amp = \langle f, h\rangle + \langle g, h\rangle \end{align*}

To verify that property (c) holds, we must check that for any scalar \(r\text{,}\) \(\langle r f, g \rangle = r \langle f, g \rangle\text{.}\) We calculate that

\begin{equation*} \langle rf, g\rangle = \int_0^1 (rf(x))g(x)\, dx = r\int_0^1 f(x)g(x)\, dx = r\langle f, g\rangle\text{.} \end{equation*}

To verify that property (d) holds, we must check that \(\langle f, f \rangle \gt 0\) if \(f \neq 0\text{.}\) But

\begin{align*} \langle f, f \rangle \amp = \int_0^1 (a_0 + a_1x)^2\; dx\\ \amp = \int_0^1 (a_0^2 + 2a_0 a_1 x + a_1^2 x^2)\; dx\\ \amp = a_0^2 + a_0 a_1 + a_1^2 / 3\\ \amp = (a_0 + a_1/2)^2 + a_1^2/12\text{.} \end{align*}

This quantity is positive unless \(a_0 + a_1/2 = 0\) and \(a_1 = 0\text{,}\) and these equations only hold simultaneously when \(a_0 = 0\) and \(a_1 = 0\text{,}\) i.e., when \(f = 0\text{.}\) Thus property (d) holds. Therefore the definition of the map \(\langle , \rangle\) defined above is an inner product.

🔗

Activity 7.2.4.

Consider \(\mathbf{P}_1\) with the inner product

\begin{equation*} \langle p, q\rangle = \int_{0}^{1} p(x)q(x)\, dx \text{.} \end{equation*}

🔗

(a)

Calculate the norm of the polynomial \(p(x) = x\text{.}\)

🔗

Solution.

We calculate that

\begin{align*} \|p\|^2 \amp = \langle p, p\rangle\\ \amp= \int_0^1 p(x)p(x)\; dx \\ \amp = \int_0^1 x^2\; dx\\ \amp = \left[ \frac{x^3}{3} \right]_0^1\\ \amp = 1/3\text{.} \end{align*}

Therefore, \(\|p\| = \sqrt{\frac{1}{3}} = \frac{1}{\sqrt{3}}\text{.}\)

🔗

(b)

Calculate the distance between \(q_1(x)=2x+1\) and \(q_2(x)=3x\text{.}\)

🔗

Solution.

We calculate that \(q_1(x) - q_2(x) = (2x+1) - 3x = -x+1\text{,}\) so

\begin{align*} \|q_1 - q_2\|^2 \amp = \langle q_1-q_2, q_1-q_2\rangle = \int_0^1 (1 - x)^2\, dx\\ \amp = \left. (-1/3) (1 - x)^3\, dx \right|_0^1\\ \amp = 1/3\text{.} \end{align*}

Therefore, \(d(q_1, q_2) = \|q_1 - q_2\| = \sqrt{\frac{1}{3}} = \frac{1}{\sqrt{3}}\text{.}\)

🔗

(c)

Find the unit vector in the same direction as \(p(x)=x\text{.}\)

🔗

Solution.

We calculated \(\|p\| = \frac{1}{\sqrt{3}}\) in Task 7.2.4.a. Therefore, the unit vector in the same direction as \(p(x)\) is the polynomial

\begin{align*} \frac{p(x)}{\|p\|} \amp = \frac{x}{\frac{1}{\sqrt{3}}}\\ \amp = \sqrt{3} \cdot x \end{align*}

🔗

(d)

Are \(x\) and \(x-2/3\) orthogonal polynomials in \(\mathbf{P}_1\text{?}\)

🔗

Solution.

To determine if the two polynomials are orthogonal, we must check whether \(\langle x, x - 2/3 \rangle = 0\text{.}\) We calculate that

\begin{align*} \langle x, x-\frac{2}{3}\rangle \amp = \int_0^1 x\left(x-\frac{2}{3}\right)\, dx\\ \amp = \int_0^1 \left(x^2 - \frac{2x}{3}\right)\, dx\\ \amp = \left. \frac{x^3}{3} - \frac{x^2}{3}\right|_0^1\\ \amp = 0 \end{align*}

🔗

So the polynomials \(x\) and \(x-2/3\) are orthogonal to one another.

🔗

Subsection Spaces of matrices

Definition 7.2.11. The Trace of a Matrix.

The trace of a square \(n \times n\) matrix \(A\text{,}\) denoted by \(\Tr(A)\text{,}\) is the sum of its diagonal entries. For \(A = [a_{ij}]\text{,}\) we have

\begin{equation*} \Tr(A) = a_{11} + a_{22} + \cdots + a_{nn}. \end{equation*}

🔗

Theorem 7.2.12.

For any \(n \times n\) matrices \(A\) and \(B\text{,}\)

\begin{equation*} \Tr(A^T) = \Tr(A) \quad\text{and}\quad \Tr(A + B) = \Tr(A) + \Tr(B)\text{.} \end{equation*}

🔗

Activity 7.2.5.

Consider \(\mathbf{M}_{22}\text{,}\) and let \(A,B\) be matrices in \(\mathbf{M}_{22}\text{.}\) Define

\begin{equation*} \langle A,B\rangle=\Tr(A^T B)\text{.} \end{equation*}

🔗

(a)

Find \(\langle A,B\rangle\) for \(A=\begin{bmatrix} 1 \amp 2 \\ 2 \amp -3\end{bmatrix}\text{,}\) \(B=\begin{bmatrix} 0 \amp 2 \\ -1 \amp 1\end{bmatrix}\text{.}\)

🔗

Solution.

We calculate that

\begin{equation*} A^T B = \begin{bmatrix} 1 \amp 2 \\ 2 \amp -3\end{bmatrix} \begin{bmatrix} 0 \amp 2 \\ -1 \amp 1\end{bmatrix} = \begin{bmatrix} -2 \amp 4 \\ 3 \amp 1\end{bmatrix} \end{equation*}

Thus

\begin{equation*} \langle A, B \rangle = \Tr(A^T B) = -2 + 1 = -1\text{.} \end{equation*}

🔗

(b)

Show that the map \(\langle , \rangle\) defined above is an inner product.

🔗

Solution.

We verify each of the four properties that define an inner product. Let \(A,B,C\) be arbitrary \(2 \times 2\) matrices:

🔗

(a) We calculate that

\begin{equation*} \Tr(A^T B) = \Tr((A^T B)^T) = \Tr(B^T (A^T)^T) = \Tr(B^T A) \end{equation*}

So \(\langle A, B \rangle = \langle B, A \rangle\text{.}\)

🔗

(b) We calculate that

\begin{equation*} \Tr((A+C)^T B) = \Tr(A^T B + C^T B) = \Tr(A^T B) + \Tr(C^T B) \end{equation*}

So \(\langle A + C, B \rangle = \langle A, B \rangle + \langle C, B \rangle\text{.}\)

🔗

\begin{equation*} \Tr(rA^T B) = r\Tr(A^T B)\text{.} \end{equation*}

So \(\langle rA, B \rangle = r \langle A, B \rangle\text{.}\)

🔗

(d) For \(A \neq 0\) with \(A = [ a_{ij} ]\text{,}\)

\begin{equation*} A^T A = [c_{ij}] \text{ where } c_{ij} = \sum_{k} a_{ki}a_{kj} \end{equation*}

The diagonal elements are: \(c_{ii} = \sum_{k} a_{ki}^2 \geq 0\text{,}\) and so

\begin{equation*} \text{Tr}(A^T A) = \sum_i \sum_k a_{ki}^2 \end{equation*}

is precisely the sum of the squares of the entries of \(A\text{,}\) which is always non-negative, and positive when at least one entry is non-zero. Thus \(\langle A, A \rangle > 0\) for \(A \neq 0\text{.}\)

🔗

(c)

Find the distance between the two matrices

\begin{equation*} A=\begin{bmatrix} 3 \amp 1 \\ 0 \amp -1\end{bmatrix} \quad\text{and}\quad B=\begin{bmatrix} -1 \amp 0 \\ 1 \amp 2\end{bmatrix}\text{.} \end{equation*}

🔗

Solution.

We recall that

\begin{equation*} d(A,B) = \|A-B\| = \sqrt{\langle A-B, A-B\rangle}\text{.} \end{equation*}

We begin by calculating

\begin{equation*} A-B = \begin{bmatrix} 3 \amp 1 \\ 0 \amp -1\end{bmatrix} - \begin{bmatrix} -1 \amp 0 \\ 1 \amp 2\end{bmatrix} = \begin{bmatrix} 4 \amp 1 \\ -1 \amp -3\end{bmatrix}\text{.} \end{equation*}

Next, we calculate that

\begin{equation*} (A-B)^T(A-B) = \begin{bmatrix} 4 \amp -1 \\ 1 \amp -3\end{bmatrix}\begin{bmatrix} 4 \amp 1 \\ -1 \amp -3\end{bmatrix} = \begin{bmatrix} 17 \amp 7 \\ 7 \amp 10\end{bmatrix}\text{.} \end{equation*}

Thus

\begin{equation*} \langle A - B, A - B \rangle = \Tr((A-B)^T (A-B)) = 17 + 10 = 27\text{.} \end{equation*}

Therefore, \(d(A,B) = \sqrt{27} = 3\sqrt{3}\text{.}\)

🔗

(d)

Find a value for \(x\) so that the two matrices

\begin{equation*} A= \begin{bmatrix} 1\amp 0\\ -2\amp 2\end{bmatrix} \quad\text{and}\quad B=\begin{bmatrix} -3\amp 2\\ x\amp 0\end{bmatrix} \end{equation*}

are orthogonal.

🔗

Solution.

We calculate that

\begin{equation*} A^T B = \begin{bmatrix} 1 \amp -2 \\ 0 \amp 2 \end{bmatrix} \begin{bmatrix} -3 \amp 2 \\ x \amp 0 \end{bmatrix} = \begin{bmatrix} -3-2x \amp 2 \\ 2x \amp 0 \end{bmatrix}\text{.} \end{equation*}

\begin{equation*} \langle A, B \rangle = \Tr(A^T B) = (-3-2x) + 0 = -3-2x = 0 \text{.} \end{equation*}

Now \(\langle A, B \rangle = 0\) precisely when \(x = -3/2\text{,}\) and so this is precisely the value of \(x\) which makes the matrices \(A\) and \(B\) orthogonal.

🔗

Prev Top Next