Let \(f\) be a scalar-valued function. Then if \(\mathbf{v}\) is a unit vector, the directional derivative of \(f\) in the direction of the vector \(\mathbf{v}\text{,}\) denoted \(D_{\mathbf{v}} f\text{,}\) is defined as
The directional derivative represents the rate of change of the function \(f\) at the point \(\mathbf{a}\) in the direction of the unit vector \(\mathbf{v}\text{.}\) For a function \(f(x,y)\) of two variables, the directional derivative \(D_{\mathbf{v}} f(\mathbf{a})\) is the slope of the tangent line to the curve obtained by intersecting the surface \(z = f(x,y)\) with the plane containing both the line passing through \(\mathbf{a}\) with direction vector \(\mathbf{v}\text{,}\) and the \(z\) axis.
Despite partial derivatives being a special case of directional derivatives, in most circumstances we can compute directional derivatives using partial derivatives, or equivalently, the gradient.
Suppose \(f\) is a function differentiable at a point \(\mathbf{a}\text{.}\) Then for each unit vector \(\mathbf{v}\text{,}\)\(D_{\mathbf{v}} f(\mathbf{a}) = \nabla f(\mathbf{a}) \cdot \mathbf{v}\text{.}\)
Define the path \(\mathbf{c}(t)=\mathbf{a}+t\mathbf{v}\) and the single-variable function \(g(t)=f(\mathbf{c}(t))=f(\mathbf{a}+t\mathbf{v})\text{.}\) Since \(\mathbf{c}(0)=\mathbf{a}\text{,}\) we have \(g(0)=f(\mathbf{a})\text{.}\) Therefore, the definition of the directional derivative gives
Write \(\mathbf{a}=(a_1,\ldots,a_n)\) and \(\mathbf{v}=(v_1,\ldots,v_n)\text{.}\) The \(i\)th coordinate of \(\mathbf{c}(t)\) is \(c_i(t)=a_i+tv_i\text{,}\) so \(c_i'(0)=v_i\text{.}\) Because \(g=f\circ\mathbf{c}\text{,}\) the chain rule (Theorem 3.5.7) gives
The greatest listed increase occurs in direction \(\mathbf v_3\text{,}\) which is aligned with the gradient. The greatest listed decrease occurs in direction \(\mathbf v_4\text{,}\) which points opposite the gradient.
Find the directional derivative of \(f\) at the point \((-1,3)\) in the direction of the vector \((-1,\sqrt{3})\text{.}\) That is, compute \(D_{\mathbf{v}} f(-1,3)\text{,}\) where \(\mathbf{v}\) is the unit vector in the direction of \((-1,\sqrt{3})\text{.}\)
The unit vector pointing in the direction from \((2,1)\) to \((3,5)\) is the unit vector pointing in the same direction as the vector \((3-2,5-1) = (1,4)\text{.}\) This vector has magnitude \(\sqrt{1^2 + 4^2} = \sqrt{17}\text{,}\) so the unit vector pointing in this direction is equal to
Now suppose \(\nabla f(\mathbf a)\neq\mathbf 0\text{,}\) and let \(\theta\) be the angle between \(\nabla f(\mathbf a)\) and a unit vector \(\mathbf v\text{.}\) By Theorem 3.6.3 and Theorem 1.1.16,
The largest possible value of \(\cos\theta\) is \(1\text{,}\) attained when \(\mathbf v\) points in the direction of the gradient. The smallest possible value is \(-1\text{,}\) attained when \(\mathbf v\) points in the opposite direction. Therefore the maximum and minimum directional derivatives are
A bowl-shaped surface is shown above the \(xy\)-plane with a point \(P_0(x_0,y_0,z_0)\) on the side of the surface. On the plane below, arrows based at the corresponding point show \(\nabla f\) for most rapid increase, \(-\nabla f\) for most rapid decrease, and perpendicular directions for zero change in \(f\text{.}\) Dashed vertical guide lines connect the point on the surface to its location in the domain.
Activity3.6.10.Finding the direction of greatest increase (U3-LO8).
Find the direction for which the directional derivative of \(f(x,y) = 3x^2 - 4xy + 2y^2\) at \((-2,3)\) is a maximum. What is the maximum value of the directional derivative?
Theorem 3.6.8 tells us that the directional derivative \(D_{\mathbf{v}} f\) is maximized when \(\mathbf{v}\) is the unit vector pointing in the direction of \((-24,20)\text{.}\) Since \((-24,20)\) has magnitude \(\sqrt{(-24)^2 + (20)^2} = \sqrt{976}\text{,}\) we see the direction which maximizes the directional derivative is
SubsectionOptimization problems and critical points
Directional derivatives describe first-order change along a chosen direction. We now use them to identify candidates for local extrema and to motivate an iterative method for decreasing a function.
Let \(f: \R^n \to \R\) be a scalar-valued function, and consider a set \(U \subseteq \R^n\text{.}\) Then we can consider two optimization problems, in which we call \(f\) the objective function of the problem, and \(U\) the feasible region of the optimization problem:
The minimization problem of \(f\) on \(U\) is the problem of finding \(\mathbf{x}^* \in U\) such that the value \(f(\mathbf{x}^*)\) is smallest among all elements of \(U\) (so the function is minimized on \(U\) by \(\mathbf{x}^*\)).
The maximization problem of \(f\) on \(U\) is the problem of finding \(\mathbf{x}^* \in U\) such that the value \(f(\mathbf{x}^*)\) is largest among all elements of \(U\) (so the function is maximized on \(U\) by \(\mathbf{x}^*\)).
Suppose \(f\) is a scalar-valued function. A point \(\mathbf{a}\) in the domain of \(f\) is a
local minimum if there is a ball centered at \(\mathbf{a}\) such that \(f(\mathbf{a}) \leq f(\mathbf{x})\) whenever \(\mathbf{x}\) is in that ball and also in the domain of \(f\text{;}\) in other words, whenever \(\mathbf{x}\) is close enough to \(\mathbf{a}\) and lies in the domain of \(f\text{.}\) In this case, we say \(f(\mathbf{a})\) is a local minimum value for the function \(f\text{.}\)
local maximum if there is a ball centered at \(\mathbf{a}\) such that \(f(\mathbf{a}) \geq f(\mathbf{x})\) whenever \(\mathbf{x}\) is in that ball and also in the domain of \(f\text{;}\) in other words, whenever \(\mathbf{x}\) is close enough to \(\mathbf{a}\) and lies in the domain of \(f\text{.}\) In this case, we say \(f(\mathbf{a})\) is a local maximum value for the function \(f\text{.}\)
global minimum if \(f(\mathbf{a}) \leq f(\mathbf{x})\) for all \(\mathbf{x}\) in the domain of \(f\text{.}\) In this case, \(f(\mathbf{a})\) is the global minimum value, or absolute minimum value, of the function \(f\text{.}\)
global maximum if \(f(\mathbf{a}) \geq f(\mathbf{x})\) for all \(\mathbf{x}\) in the domain of \(f\text{.}\) In this case, \(f(\mathbf{a})\) is the global maximum value, or absolute maximum value, of the function \(f\text{.}\)
The graph of the function \(-f\) is obtained by reflecting the graph of the function \(f\) in the output axis. This operation turns local and global maxima into local and global minima, and vice versa. So methods for finding minima for a general function often give methods for finding maxima, which makes the two problems often equivalent.
Suppose \(f\) is a scalar-valued function that is differentiable at a point \(\mathbf{a}\text{.}\) Then we say \(\mathbf{a}\) is a critical point if \(\nabla f(\mathbf{a}) = \mathbf{0}\text{.}\)
For a scalar-valued function of two variables, for example, \(f(x,y)\text{,}\) a critical point is a solution to the two equations \(f_x(x,y) = 0\) and \(f_y(x,y) = 0\text{.}\)
Just as the derivative of a scalar-valued function of a single variable helps identify local minima and maxima, partial derivatives provide information about possible local minima/maxima.
The critical points are the pairs \((x,y)\) that simultaneously solve the equations \(2x - 2 = 0\) and \(2y + 4 = 0\text{.}\) So the only critical point is \((1, -2)\text{.}\)
We compute that \(f_x(x,y) = 2x\) and \(f_y(x,y) = -2y\text{.}\) The only \((x,y)\) such that \(f_x(x,y) = 2x = 0\) and \(f_y(x,y) = -2y = 0\) is \((0,0)\text{.}\) This is the only critical point.
The equation \(\nabla f=0\) finds candidates for local extrema. It does not by itself find boundary extrema, constrained extrema, or absolute extrema. Always combine critical-point information with the domain and boundary conditions.
Using the calculus we have developed, it is usually straightforward to write the equations that critical points must satisfy. Activity 3.6.20 also shows the problem: those equations may be nonlinear and difficult or impossible to solve exactly. This is different from the linear systems in Unit 2, where Gaussian elimination gave a systematic exact method.
When the critical-point equations cannot be solved directly, one often uses a numerical method. By Theorem 3.6.8, the negative gradient gives the direction of steepest local decrease.
Unless the gradient is zero, the local model predicts a decrease. This is a local prediction, not a convergence theorem. It motivates the gradient-descent update
Move in the direction of the negative gradient: set \(\mathbf{x}_1 = \mathbf{x}_0 - \alpha_0 \nabla f(\mathbf{x}_0)\text{,}\) where \(\alpha_0 > 0\) is a step size (or learning rate) parameter.
Repeat steps 2-3 with step sizes \(\alpha_1, \alpha_2, \dots\) to successively define points \(\mathbf{x}_2 = \mathbf{x}_1 - \alpha_1 \nabla f(\mathbf{x}_1)\text{,}\)\(\mathbf{x}_3 = \mathbf{x}_2 - \alpha_2 \nabla f(\mathbf{x}_2)\text{,}\) and so on, with the hope that these points converge to a local minimum.
Intuitively, gradient descent works because each successive value \(\mathbf{x}_k\) is obtained by shifting the previous value \(\mathbf{x}_{k-1}\) in the direction in which the function \(f\) decreases the fastest. Gradient descent may converge to a local minimum under additional assumptions on the function and the learning rates. The examples below illustrate possible behavior; they do not prove convergence.
The learning rate is alpha. The vector grad_f(x) must have the same shape as x, because the update subtracts one vector from another. If the minus sign were a plus sign, the update would move in the direction of steepest increase rather than steepest decrease.
using gradient descent. Use the starting point \(\mathbf{x}_0 = (0,0)\text{,}\) a constant step size of \(0.1\text{,}\) and stop after computing \(\mathbf{x}_3\text{.}\) You may use a calculator to perform these computations.
and classify it after developing second-order tools. Thus \(\mathbf x_3\) is not yet a good approximation. If we continue the iteration, the points \(\mathbf x_k\) for, say, \(k\geq1000\) are very close to that critical point.
The figure shows a side view of the surface \(z=f(x,y)\) for \(f(x,y)=x^2-3xy+3y^2+5y+2x\) over a small region near the origin. Four red points labeled \(\mathbf{x}_0\text{,}\)\(\mathbf{x}_1\text{,}\)\(\mathbf{x}_2\text{,}\) and \(\mathbf{x}_3\) are connected by red line segments on the surface, showing the first three gradient descent steps from \((0,0)\text{.}\)
Figure3.6.24.Initial steps of gradient descent for \(f(x,y)=x^2-3xy+3y^2+5y+2x\) with starting point \(\mathbf{x}_0 = (0, 0)\text{.}\) For \(k=1,2,3\text{,}\) the point \(\mathbf{x}_k\) is determined by the rule \(\mathbf{x}_k = \mathbf{x}_{k-1} - (0.1) \nabla f(\mathbf{x}_{k-1})\text{.}\) Adapted from Stanford’s MATH 51 textbook.
