View source for Calculus/Approximating Values of Functions

{{Calculus/Top Nav|Euler's Method|Differentiation/Applications of Derivatives/Exercises}}

Although many times, a value can and does have an exact representation that is and can be described by some function, it is sometimes useful to get an approximation of those values especially in many real world contexts. For example, a construction worker might need a room to be <math>\sqrt{2}</math> feet long. However, this value is not useful because most rulers do not have a measurement for <math>\sqrt{2}</math>. Instead, workers need an approximation of the length to be able to construct a room that is <math>\sqrt{2}</math> feet long.

Some numbers were and still are very hard to approximate, but calculus makes it easier to do so. The subfield of [[w:Numerical_analysis|Numerical Analysis]] studies algorithms used to approximate numbers, including but not limited to the residual (how much the value is off from the true value), the level of decimal precision, and the amount of times the procedure needs to be done to reach a certain level of precision.

While this section will not be a replacement for numerical analysis (not even close), this section will hopefully introduce efficient algorithms to approximate values to surprising levels of accuracy.

Before diving into this section, Section {{Calculus/map page|Continuity}} already introduced a method to approximate the solution to a function using calculus as justification behind this approximation algorithm, referred to as the Bisection Method. Using calculus to approximate values should therefore not be very surprising.

== Linear Approximation ==
[[File:Tangent Line Approximation.svg|thumb|318x318px|'''Figure 1''': The tangent line to the function at <math>\alpha</math> approximates the <math>f(c)</math> quite well.]]


Recall one of the definitions of the derivative: it is the slope of the tangent line at a point, <math>x = \alpha</math>, of the function <math>f</math>. Thinking about the local behavior of the function <math>f</math> around <math>x = \alpha</math>, the tangent line can be a good approximation of the value <math>f(c)</math> (refer to '''Figure 1''') if <math>c-\alpha</math> is small '''and''' <math>f(c)-f(\alpha)</math> is small.

{{Calculus/Def|title=Linear Approximation|
text=If <math>f(\alpha)</math> is a known value for the differentiable and continuous function <math>y = f(x)</math>, the derivative at that point is <math>f^\prime (\alpha)</math>, and <math>|f(c) - f(\alpha)| = \epsilon_0</math> for all small <math>\epsilon_0 > 0</math>, then the value of <math>f(c)</math> on the interval <math>\left(x - \delta, x + \delta\right)</math> for some small <math>\delta > 0</math> is approximated by the following equation:
{{Equation|<math>f(c) \approx f^\prime (\alpha) (c - \alpha) + f(\alpha)</math>|1|center}}}}

'''Justification''': Notice the tangent line at <math>x = \alpha</math> for some differentiable function <math>f(x)</math> is given by the following equation:

{{Equation|<math>h(x) = f^\prime (\alpha) (x - \alpha) + f(\alpha)</math>|2}}

where <math>h(x)</math> is the equation of the tangent line.

If we are trying to obtain <math>f(c)</math> (the true value) through the tangent line, and the distance <math>|f(c) - f(\alpha)| = \epsilon_0</math> for all small <math>\epsilon_0 > 0</math>, then <math>h(c) \approx f(c)</math>. Therefore,
: <math>f(c) \approx h(c) = f(\alpha) + f^\prime (\alpha) (c - \alpha) \qquad \square</math>

Notice that for this technique to be used, it needs to be the case that
# <math>f(x)</math> is differentiable at <math>x = \alpha</math> and continuous in <math>\left[\alpha, c\right]</math>.
# <math>|c-\alpha|</math> is small '''and''' <math>|f(c)-f(\alpha)|</math> is small. Otherwise, some very strange possible approximations may appear.
# <math>f(x)</math> is [[w:Monotonic_function|monotonic]] in <math>\left[\alpha, c\right]</math>. (You will learn this more comprehensively in Section {{Calculus/map page|Sequences}}.)
If any one of these conditions are false, then this technique will either not work or will not be very useful.

{{ExampleRobox|title=Example {{Calculus/map page|Approximating Values of Functions}}.1: Approximating <math>\sqrt{1.01}</math>|theme=3}}
Let <math>f(x) = \sqrt{x}</math>. The exact value of <math>\sqrt{1.01} = f(1.01)</math>.
: <math>f^\prime (x) = \frac{1}{2\sqrt{x}}</math>
The tangent line equation at <math>x = 1</math> is given by
: <math>y - f(1) = f^\prime (1) (x - 1) \Leftrightarrow y = f^\prime (1) (x - 1) + f(1)</math>
: <math>y = \frac{1}{2}x + \frac{1}{2}</math>
Let <math>g(x) = y</math>. Suppose <math>g(1.01) \approx f(1.01)</math>.

Then, <math>f(1.01) \approx g(1.01) = \frac{1}{2}(1.01) + \frac{1}{2} = 1.005</math>. Therefore, <math>\sqrt{1.01} \approx 1.005</math>. <math>\blacksquare</math>
{{Robox/Close}}

The benefit of linear approximation is that instead of having to use harder to understand functions, we estimate the value of a function using linear functions, arguably the easiest possible calculation we will ever have to use, assuming the value of the derivative is easy to find.

=== Determining over- or underestimation ===

Notice that for any approximation we obtain using a '''tangent line approximation''' (same as a linear approximation), there exists a remainder term that will make it equal to the true value we can obtain from the function. That is,

{{Equation|<math>f(c) = f(\alpha) + f^\prime (\alpha) (c - \alpha) + R_2</math>|3|center}}

where <math>R_2</math> is the remainder term. This, unfortunately, does not give us a precise estimate of the residual, especially if we cannot find the exact value of the term. While there is a technique to determine the upper bound of a residual for this type of estimate, it will be done much later in Section {{Calculus/map page|Taylor series}}.

The best solution we have for now is determining if the estimate we have given is below or above the true value, which can be done with the following technique.

{{Calculus/Def|title=Above or Below the True Value of Tangent Line Approximation|
text=
* If <math>f</math> is concave down in the interval between <math>\alpha</math> and <math>c</math>, the approximation will be an overestimate.
* If <math>f</math> is concave up in the interval between <math>\alpha</math> and <math>c</math>, the approximation will be an underestimate.}}

'''Justification''': Suppose <math>f(x)</math> is a twice differentiable function in <math>\left[\alpha, c\right]</math> and <math>\alpha < c</math>.
: '''Case 1(A)''': Let <math>f^\prime (\alpha) > 0</math> and <math>f^{\prime\prime} (x_0) < 0</math> for <math>x_0\in\left[\alpha, c\right]</math>. Then, the tangent line, <math>h(x)</math> at <math>(\alpha, f(\alpha))</math> has positive slope and <math>h(c)>f(c)</math> (see the  bottom function of '''Figure 1''').
: '''Case 1(B)''': Let <math>f^\prime (\alpha) > 0</math> and <math>f^{\prime\prime} (x_0) > 0</math> for <math>x_0\in\left[\alpha, c\right]</math>. Then, the tangent line, <math>h(x)</math> at <math>(\alpha, f(\alpha))</math> has positive slope and <math>h(c)<f(c)</math>.
: '''Case 2(A)''': Let <math>f^\prime (\alpha) < 0</math> and <math>f^{\prime\prime} (x_0) < 0</math> for <math>x_0\in\left[\alpha, c\right]</math>. Then, the tangent line, <math>h(x)</math> at <math>(\alpha, f(\alpha))</math> has negative slope and <math>h(c)>f(c)</math>.
: '''Case 2(B)''': Let <math>f^\prime (\alpha) < 0</math> and <math>f^{\prime\prime} (x_0) > 0</math> for <math>x_0\in\left[\alpha, c\right]</math>. Then, the tangent line, <math>h(x)</math> at <math>(\alpha, f(\alpha))</math> has negative slope and <math>h(c)<f(c)</math> (see the  bottom function of '''Figure 1''').
Since a concave down function has <math>h(c) > f(c)</math> no matter the slope of the tangent, and a concave up function has <math>h(c) < f(c)</math> no matter the slope of the tangent, we have justified what we wanted to do. <math>\square</math>

{{ExampleRobox|title=Example {{Calculus/map page|Approximating Values of Functions}}.2: Is <math>\sqrt{1.01} \approx \frac{1}{2}(1.01) + \frac{1}{2} = 1.005</math> an over- or underestimate?|theme=3}}
Let <math>f(x) = \sqrt{x}</math>.
: <math>f^\prime (x) = \frac{1}{2} x^{-\frac{1}{2}} = \frac{1}{2\sqrt{x}}</math>
: <math>f^{\prime\prime} (x) = -\frac{1}{2} x^{-\frac{3}{2}}</math>
The function is concave down for all <math>x \ge 1</math> because
: <math>\begin{align}
 f^{\prime\prime} (x) &= -\frac{1}{2} (x)^{-\frac{3}{2}} \\
 &= -\frac{1}{2x^{3/2}}
\end{align}</math>.
: <math>\Rightarrow -\frac{1}{2} \le -\frac{1}{2x^{3/2}} < 0</math> for all <math>x \ge 1</math>.
The last implication is justified because <math>\lim_{x\to\infty} f^{\prime\prime} (x) = 0</math> (as an exercise, show this yourself). If <math>f^{\prime\prime} (x)</math> is [[w:Monotonic_function|monotonically]] decreasing and bounded (which it is – you can show this multiple ways), then we can be certain that the inequality shown is correct.

Because <math>f(x)</math> is certainly concave down from <math>x = 1</math> to <math>x = 1.01</math>, <math>\sqrt{1.01} \approx 1.005</math> is an overestimate of the true value. <math>\blacksquare</math>
{{Robox/Close}}

=== Issues with Linear Approximation ===
While linear (or tangent line) approximation is a powerful, easy tool that can be used to approximate functions, it does have its issues. These issues were alluded to when introducing the technique. Each issue will highlight why this tool may not be very useful all the time.

{{ExampleRobox|title=Example {{Calculus/map page|Approximating Values of Functions}}.3: Approximate <math>\sqrt[3]{0.01}</math> using Tangent Line Approximation|theme=3}}
Let <math>f(x) = \sqrt[3]{x} = x^{1/3}</math>. Since we are approximating <math>f(0.01)</math>, we may use the tangent line approximation at <math>x=0</math> to obtain an approximation at <math>f(0.01)</math>. Unfortunately, <math>f^\prime(0) = \lim_{x\to 0} \dfrac{\sqrt[3]{x}}{x}</math> does not converge to a finite value. This therefore means that the tangent line is either vertical or does not exist. Since the derivative does not exist at <math>x = 0</math>, it is impossible to obtain an accurate approximation using the linear method. <math>\blacksquare</math>
{{Robox/Close}}

{{ExampleRobox|title=Example {{Calculus/map page|Approximating Values of Functions}}.4: Approximate <math>(-0.8)^4 - 1.5 (-0.8)^2</math> using Tangent Line Approximation|theme=3}}
Let <math>f(x) = x^4 - \frac{3}{2} x^2</math>. Since we are approximating <math>f(-0.8)</math>, and <math>-0.8 - (-1) = 0.2</math> is a small difference in <math>x</math>, we may use the tangent line approximation at <math>x=-1</math> to obtain an approximation at <math>f(-0.8)</math>.
: <math>f(-1) = (-1)^4 - 1.5 (-1)^2 = 1 - 1.5 = -0.5</math>
: <math>f^\prime (x) = 4x^3 - 3x</math>
The tangent line equation at <math>x = -1</math> is given by
: <math>y - f(-1) = f^\prime (-1) (x + 1) \Leftrightarrow y = f^\prime (-1) (x + 1) + f(-1)</math>
: <math>y = -x - 1.5</math>
Let <math>g(x) = y</math>. Suppose <math>g(-0.8) \approx f(-0.8)</math>.

Then, <math>f(-0.8) \approx g(-0.8) = -(-0.8) - 1.5 = -0.7</math>. Therefore, <math>(-0.8)^4 - 1.5 (-0.8)^2 \approx -0.7</math>.

However, this approximation is very off from the actual value. It has an error of <math>R_2 = 0.1496</math> within the actual value of the function at <math>x=-0.8</math>. The reason for this large error has to do with something subtle.

While the derivative does exist at <math>\alpha = -1</math>, the function is not monotonic from <math>\left[-1,-0.8\right]</math>: the function will both decrease and increase. Recall from your reading that a function is monotonically decreasing if and only if for any <math>x_1, x_2 \in \left[a,b\right]</math> and <math>x_1 < x_2</math>, <math>h(x_1) > h(x_2)</math>. However, there exists a counterexample for <math>f(x)</math> above, which becomes apparent [https://www.desmos.com/calculator/b6eygpgmgn if you graph the function].

Hence, it would be irresponsible to use a tangent line to approximate the value of <math>(-0.8)^4 - 1.5 (-0.8)^2</math>. <math>\blacksquare</math>
{{Robox/Close}}

Because this will be something you will learn more comprehensively in Section {{Calculus/map page|Sequences}}, assume that all exercises have a monotonic function. This example is simply meant to illustrate a common pitfall that this method falls to.

== Newton-Raphson Method ==
While tangent line approximations are very helpful, they tend to only be useful when you know a nearby value. However, if there does not exist a nearby value to help estimate its value, then it will be very difficult to obtain a precise estimate of the exact value desired.

The Newton-Raphson method, introduced in Section {{Calculus/map page|Newton's Method}}, is a useful method to determine the zeros of a function, whether polynomial, transcendental, irrational, exponential, etc. However, the Newton-Raphson method can also be used to approximate values of specific functions.

{{Calculus/Def|title=Newton-Raphson Method|
text=Suppose <math>\alpha</math> is the value of interest. Let <math>x = \alpha</math>. There exists some function <math>f(x)</math> so that <math>f(\alpha) = 0</math>. If an approximation of <math>\alpha</math> is desired, then iteratively find <math>\alpha</math> as follows:
{{Equation|<math>\alpha \approx x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}</math>|4|center}}}}

If you read Section {{Calculus/map page|Newton's Method}}, then this equation should be known and already justified to you.

{{ExampleRobox|title=Example {{Calculus/map page|Approximating Values of Functions}}.5: Approximate <math>\sqrt{72}</math>.|theme=3}}
Let <math>x = \sqrt{72}</math>. We need to manipulate this equation so that we may obtain <math>f(\alpha) = 0</math>.
: <math>\begin{align}
 x &= \sqrt{72} \\
 x^2 &= 72 \\
 x^2 - 72 &= 0 \\
\end{align}</math>
From this, let <math>f(x) = x^2 - 72</math>. Since <math>x=\sqrt{72}</math> is a root of the equation, we can use the Newton-Rhapson method to approximate <math>\sqrt{72}</math>. First, we choose an initial guess. Since <math>f(8) = -8 < 0 < f(9) = 9</math>, and <math>f(9) > \left| f(8) \right| > 0</math>, <math>x_0 = 8</math> is a good initial guess. Before we can begin, we need the derivative of the function, which can be easily shown to be <math>f^\prime (x) = 2x</math>. Now we begin by finding the root.
: <math>\begin{align}
 x_1 &= 8 - \frac{-8}{2\cdot 8} &= 8.5 \\
 x_2 &= 8.5 - \frac{8.5^2 - 72}{2\cdot 8.5} &= 8.47058823529 \\
 x_3 &= 8.47058823529 - \frac{8.47058823529^2 - 72}{2 \cdot 8.47058823529} &= 8.48529411765
\end{align}</math>
Out of convenience, we will stop here. However, the value we have obtained is already correct to ''four'' decimal places. <math>\blacksquare</math>
{{Robox/Close}}

{{ExampleRobox|title=Example {{Calculus/map page|Approximating Values of Functions}}.6: Approximate <math>\sqrt{2} + \sqrt{3}</math>.|theme=3}}
Let <math>x = \sqrt{2} + \sqrt{3}</math>. We need to manipulate this equation so that we may obtain <math>f(\alpha) = 0</math>. Keep in mind to eliminate as many square roots as possible so that our jobs are easier.
: <math>\begin{align}
 x &= \sqrt{2} + \sqrt{3} \\
 x^2 &= 2 + 2\sqrt{6} + 3 \\
 x^2 - 5 &= 2\sqrt{6} & \\
 x^4 - 10x^2 + 25 &= 4 \cdot 6 \\
 x^4 - 10x^2 + 1 &= 0
\end{align}</math>
Notice this equation has four possible roots, but we only care about one of them. Looking back:
:<math>1 + 1  = 2 < \sqrt{2} + \sqrt{3} < 4 = 2 + 2 </math>
Notice that <math>f(3) = -8 < 0 < f(4) = 97</math>, so we choose <math>x_0 = 3</math>.
: Let <math>f(x) = x^4 - 10x^2 + 1</math>. Then, <math>f^\prime (x) = 4x^3 - 20x</math>.
Notice that <math>4x(x^2 - 5) = 0 \Rightarrow x=0 \vee x = \pm\sqrt{5}</math> means that <math>f^\prime (x) \ne 0</math> for <math>x\in\left(\sqrt{5},4\right)</math>.

Finally, notice that <math>x_{n+1} = x_n - \frac{x_n^4 - 10x_n^2 + 1}{4x_n^3 - 20x_n}</math> cannot be simplified. We are going to have to work with these values. (Keep in mind that even without calculators, many people in the past used [[w:Slide_rule|slide rulers]] to work with these ugly values. We have the benefit of an electric computer on our fingertips as opposed to an analog one.)

Now we begin by finding the root.
: <math>\begin{align}
 x_1 &= 3 - \frac{-8}{48} = \frac{19}{6} &\approx 3.16666666667 \\
 x_2 &= \frac{19}{6} - \dfrac{\frac{1657}{1296}}{\frac{3439}{54}} = \frac{86569}{27512} &\approx 3.14659057866 \\
 x_3 &= 3.14659057866 - \frac{0.0201173085466}{61.686167862} &\approx 3.14626445516
\end{align}</math>
Out of convenience, we will stop here. However, the value we have obtained is already correct to ''six'' decimal places. <math>\blacksquare</math>
{{Robox/Close}}

=== Failure Analysis ===
[[Image:NewtonsMethodConvergenceFailure.svg|thumb|300px|'''Figure 2''': The tangent lines of <math>x^3 - 2x + 2</math> at <math>0</math> and <math>1</math> intersect the <math>x</math>-axis at <math>1</math> and <math>0</math> respectively, illustrating why Newton's method oscillates between these values for some starting points.]]
Of course, as we know from Section {{Calculus/map page|Newton's Method}}, the Newton-Raphson method is not perfect and will fail in some instances. One obvious instance is when the derivative at a particular point is zero. Because that is in the denominator, we cannot find the next possible root once that occurs. However, there are [[w:Newton's Method#Failureanalysis|others]].

==== Starting Point Enters a Cycle ====
For some functions, some starting points may enter an infinite cycle, preventing convergence. Let

:<math>f(x) = x^3 - 2x + 2 \!</math>

and take <math>0</math> as the starting point. The first iteration produces <math>1</math> and the second iteration returns to <math>0</math> so the sequence will alternate between the two without converging to a root (see '''Figure 2'''). The real solution of this equation is <math>-1.76929235\ldots</math>. In those instances, one should select another starting point.

==== Derivative does not exist at root ====
A simple example of a function where Newton's method diverges is trying to find the cube root of zero. The cube root is continuous and infinitely differentiable, except for <math>x = 0</math>, where its derivative is undefined:

:<math>f(x) = \sqrt[3]{x}.</math>

For any iteration point <math>x_n</math>, the next iteration point will be:

:<math>x_{n+1} = x_n - \frac{f(x_n)}{f^\prime (x_n)} = x_n - \frac{{x_n}^\frac13}{\frac13{x_n}^{-\frac23}} = x_n - 3x_n = -2x_n.</math>

The algorithm overshoots the solution and lands on the other side of the <math>y</math>-axis, farther away than it initially was; applying Newton's method actually doubles the distances from the solution at each iteration.

In fact, the iterations diverge to infinity for every <math>f(x) = |x|^\alpha</math>, where <math>0 < \alpha < \frac{1}{2}</math>. In the limiting case of <math>\alpha = \frac{1}{2}</math>, the iterations will alternate indefinitely between points <math>x_0</math> and <math>-x_0</math>, so they do not converge in this case either.

==== Discontinuous Derivative ====
If the derivative is not continuous at the root, then convergence may fail to occur in any neighborhood of the root. Consider the function
:<math>f(x) = \begin{cases}
 0 & \text{if } x = 0\\
 x + x^2 \sin \left(\dfrac2x\right) & \text{if } x \ne 0
\end{cases}</math>
:<math>f^\prime (x) = \begin{cases}
 0 & \text{if } x = 0\\
 1 + 2x \sin \left(\dfrac2x\right) - 2\cos\left(\dfrac2x\right) & \text{if } x \ne 0
\end{cases}</math>
Within any neighborhood of the root, this derivative keeps changing sign as <math>x</math> approaches <math>0</math> from the right (or from the left) while <math>f(x) \ge x - x^2 > 0</math> for <math>0 < x < 1</math>.

Thus, <math>\dfrac{f(x)}{f^\prime (x)}</math> is unbounded near the root, and Newton's method will diverge almost everywhere in any neighborhood of it, even though:
* the function is differentiable (and thus continuous) everywhere;
* the derivative at the root is nonzero;
* <math>f</math> is infinitely differentiable except at the root; and
* the derivative is bounded in a neighborhood of the root (unlike <math>f(x)/f^\prime (x)</math>).

== Euler's Method ==
As will be shown in the examples below, Euler's method does not tend to an approximation as quickly as Newton's method. There are a number of reasons for this, but Euler's method does not stop being useful in spite of this issue. The examples will showcase how useful Euler's method can be however.

{{ExampleRobox|title=Example {{Calculus/map page|Approximating Values of Functions}}.7: Approximate <math>\sqrt{6}</math>.|theme=3}}
Let <math>y = \sqrt{x}</math>. We need to find a differential equation <math>y^\prime = f(x,y)</math>. Notice that we will do something in the last step which will be important for approximating <math>y(6)</math>.
: <math>\begin{align}
 y(x) &= \sqrt{x} \\
 y^\prime (x) &= \frac{1}{2\sqrt{x}} \\
 y &= \frac{1}{2y(x)}
\end{align}</math>
The last step shown here is very important. We need to approximate <math>\sqrt{x}</math>. However, if we are trying to approximate <math>\sqrt{x}</math>, that cannot be done if we need to use the square root to do so. Instead, since <math>y = \sqrt{x}</math> and Euler's method gives us new <math>y</math>-values to work with, we can get an approximation.

We next need to choose an initial point, <math>(x_0,y_0)</math>. This is the only point in which we use the actual value from the function <math>y(x)</math>. In this instance, since we want <math>\sqrt{6}</math>, we will choose <math>y(4) = 2</math>. From there, we can choose the step size, which can be arbitrary. Here, we will choose <math>\Delta x_{\rm step} = 1</math>
: <math>\begin{array}{l || c|c || c}
	n & x_{n} = x_{n-1} + \Delta x_{\rm step} & y_{n} = y_{n-1} + \Delta x_{\rm step} \cdot f(x_{n-1},y_{n-1}) & f(x_n,y_n) \\
    \hline
	1 & 4 & 2.00000 & \frac{1}{2 \cdot 2.00} = 0.25000 \\
	2 & 5 & 2.25000 & \frac{1}{2 \cdot 2.25} = 0.22222 \\
    3 & 6 & 2.47222 & \frac{1}{2 \cdot 2.47222}
\end{array}</math>
As can be seen here, the approximation for <math>\sqrt{6}</math> is <math>2.47222</math>. This has an absolute error of <math>0.0227325</math>, which is pretty good for most contexts. If this was to design a box for example, then this is an okay margin of error. <math>\blacksquare</math>
{{Robox/Close}}

While the margin of error appears to be big compared to the local linear approximation of <math>\sqrt{1.01}</math> (with an absolute error of <math>0.0000124</math>), keep in mind that our step size is considerably larger than the one shown in Example {{Calculus/map page|Approximating Values of Functions}}.1. If we have chosen a smaller step size, then the approximation would be more accurate (if only a little more cumbersome). For instance, with a step size of <math>\Delta x_{\rm step} = 0.25</math>, the absolute error would be approximately <math>0.005</math>.

{{Calculus/Top Nav|Euler's Method|Differentiation/Applications of Derivatives/Exercises}}
{{Calculus/TOC}}
{{Calculus/Stub}}
{{BookCat}}