A Clean Introduction to Convex Optimization
Introduction
Among all the benchmark functions used in numerical optimization, the Sphere function stands out for its simplicity. It is smooth, strictly convex, and has a single global minimum — making it the perfect starting point for understanding how optimization algorithms behave before tackling more complex, non-convex landscapes. In this article, we implement and visualize the minimization of the Sphere function in two dimensions, comparing a hand-written gradient descent routine against SciPy’s BFGS optimizer.
Mathematical Definition
The Sphere function in $n$ dimensions is defined as:
$$
f(\mathbf{x}) = \sum_{i=1}^{n} x_i^2
$$
For our two-dimensional example, this simplifies to:
$$
f(x_1, x_2) = x_1^2 + x_2^2
$$
Because the function is a simple sum of squares, its gradient has a closed-form expression:
$$
\nabla f(\mathbf{x}) = 2\mathbf{x}
$$
and its Hessian matrix is constant and positive definite:
$$
H(\mathbf{x}) = 2I
$$
where $I$ is the identity matrix. This positive-definite Hessian is exactly what makes the Sphere function strictly convex, guaranteeing that any local minimum found is also the global minimum, located at $\mathbf{x}^* = \mathbf{0}$ with $f(\mathbf{x}^*) = 0$.
Python Implementation
The code below defines the Sphere function and its gradient, runs a custom gradient descent optimizer, cross-validates the result with SciPy’s BFGS method, and visualizes both optimization paths on a 3D surface, a contour map, and a convergence plot.
1 | import numpy as np |
Code Explanation
Function and gradient definitions. sphere() computes $\sum x_i^2$ using np.sum(x ** 2), which works for any dimensionality. sphere_gradient() returns $2\mathbf{x}$ directly, avoiding the need for numerical differentiation and giving both optimizers an exact, noise-free gradient signal.
Custom gradient descent. The gradient_descent() function implements the simplest possible first-order optimizer: at each step it moves in the direction opposite to the gradient, scaled by a fixed learning_rate. Because the Sphere function’s Hessian is $2I$, the update rule simplifies to $x_{k+1} = x_k(1 - 2\eta)$, where $\eta$ is the learning rate. With $\eta = 0.25$, this factor equals $0.5$, so the distance to the origin is halved at every iteration — a textbook example of linear convergence. The loop also includes an early-stopping condition based on gradient norm, so it won’t run needless iterations once it’s essentially converged.
BFGS reference. SciPy’s minimize() with method='BFGS' is a quasi-Newton method that builds an approximate Hessian from gradient information as it iterates. Since the true Hessian of the Sphere function is a constant multiple of the identity, BFGS converges extremely fast — typically within just a handful of steps. The callback argument lets us record every intermediate point xk, which we later plot alongside the gradient descent path.
Why no separate “fast” version is needed. Because the Sphere function and its gradient are trivially cheap to evaluate (just squaring and summing a handful of numbers), this problem never becomes a computational bottleneck even at high iteration counts. The vectorized NumPy operations already run in microseconds per step, so no further speed optimization is necessary here — this keeps the code simple and readable.
Visualization and Interpretation
The figure produced by the script above contains three panels side by side:
3D Surface Plot (left). This shows the paraboloid shape of the Sphere function, colored with the
plasmacolormap. The cyan trajectory traces the gradient descent path as it spirals down toward the bowl’s bottom, visually confirming the smooth, funnel-like geometry that makes this function so easy to optimize.Contour Plot (center). Viewed from directly above, the concentric circles represent level sets of equal function value. Both optimization paths are overlaid: the cyan markers show gradient descent taking small, steady steps, while the magenta squares show BFGS reaching the center in dramatically fewer steps thanks to its curvature-aware search direction.
Convergence Plot (right). Plotted on a logarithmic y-axis, this panel makes the difference in convergence speed unmistakable. Gradient descent’s function value decreases in a straight line on the log scale (confirming the theoretical linear/geometric convergence rate), while BFGS’s curve drops almost vertically, reaching machine-precision accuracy in only a few iterations.
The console output — showing the starting point, final coordinates, function values, and iteration counts for both methods — should also confirm that both optimizers converge to the same global minimum at $[0, 0]$ with $f(\mathbf{x}) \approx 0$.
============================================================ Sphere Function Minimization Summary ============================================================ Starting point : [4. 4.5] Gradient Descent result : [1.45519152e-11 1.63709046e-11], f = 4.798e-22, iterations = 38 BFGS result : [-6.66133815e-16 0.00000000e+00], f = 4.437e-31, iterations = 3 True global minimum : [0. 0.], f = 0.0 ============================================================
Conclusion
The Sphere function may be the “hello world” of optimization problems, but it offers genuine insight: it isolates the raw behavior of an algorithm’s convergence rate without the confounding effects of non-convexity, saddle points, or multiple local minima. By comparing a from-scratch gradient descent implementation against SciPy’s BFGS, we get a clear, visual demonstration of why second-order (curvature-aware) methods so dramatically outperform first-order methods on well-conditioned convex problems — a lesson that carries directly into far more complex optimization landscapes.
