Taylor Series Expansion: A Local Lens for Functions
Taylor Series Expansion: A Local Lens for Functions
6 min read
Taylor series is one of the most useful ideas in applied math: near a point, a complicated function behaves like a polynomial. That local approximation is exactly what powers Newton-style optimization, uncertainty propagation, and many numerical methods.
Related Posts:
- From Gradients to Hessians - Why first- and second-order terms govern optimization behavior
- The Evolution of Optimization - Where local approximations fit in the broader optimization timeline
- Why Intersection Fails in Lagrange Multipliers - Gradient geometry at constrained optima
Core Idea
For a smooth function $f(x)$, the Taylor expansion around $x=a$ is:
\[f(x) = f(a) + f'(a)(x-a) + \frac{f''(a)}{2!}(x-a)^2 + \cdots + \frac{f^{(n)}(a)}{n!}(x-a)^n + R_n(x).\]- The constant term sets the baseline.
- The linear term gives slope (first-order behavior).
- The quadratic term gives curvature (second-order behavior).
- Higher-order terms refine the approximation farther from $a$.
When $a=0$, this is called the Maclaurin series.
Three Expansions You Use All the Time
Around $x=0$:
\[e^x = 1 + x + \frac{x^2}{2!} + \frac{x^3}{3!} + \cdots\] \[\sin x = x - \frac{x^3}{3!} + \frac{x^5}{5!} - \cdots\] \[\ln(1+x) = x - \frac{x^2}{2} + \frac{x^3}{3} - \cdots \quad (\mid x \mid < 1).\]These are not just textbook formulas; they are practical approximations in models, solvers, and error analysis.
Why the Remainder Matters
A Taylor polynomial is only trustworthy if the remainder is small. For first-order and second-order approximations:
\[f(a+h) \approx f(a) + f'(a)h\] \[f(a+h) \approx f(a) + f'(a)h + \frac{1}{2}f''(a)h^2\]The second form is usually much better when curvature is significant. In optimization, this is exactly why Hessian information can dramatically improve step quality.
Why It Matters in ML and Vision
- Optimization updates: First-order methods use gradient terms; Newton and quasi-Newton methods use second-order structure from Taylor approximations.
- Loss landscape intuition: Near critical points, the quadratic term explains minima, maxima, and saddles.
- Numerical stability: Many algorithms approximate nonlinear functions locally before solving.
If you remember one sentence: Taylor series is the bridge from nonlinear functions to tractable local models.