Minimizing Potential Energy in a Two-Variable System

August 9, 2026

A Spring Chain Model in Python

Physical systems love to settle into their lowest-energy configuration. A ball rolls to the bottom of a valley, a molecule relaxes into its most stable bond lengths, a chain of springs stretches until every force balances out. This principle — energy minimization — is one of the most powerful ideas in physics and computational chemistry, and it’s a beautiful playground for numerical optimization.

In this article, we’ll build a concrete example: a 1D chain of springs connecting four particles, where the two end particles are fixed and the two middle particles are free to move. This gives us exactly two variables to optimize — the positions of the two free particles — and it’s a simplified but genuine model of how molecular mechanics software finds the equilibrium geometry of a molecule (think of it as a toy triatomic chain, like a simplified CO₂ backbone).

We’ll solve it three different ways (gradient descent, scipy’s optimizer, and an exact analytical solution), visualize the energy landscape in 3D, and compare a slow vs. a fast implementation.

1. The Physical Model

Imagine four particles arranged along a line, connected in series by three springs:

$P_0$ is fixed at $x_0 = 0$
$P_3$ is fixed at $x_3 = 12$
$P_1$ (position $x_1$) and $P_2$ (position $x_2$) are free — these are our two variables

Each spring has its own stiffness $k_i$ and natural (rest) length $L_i$. The total potential energy of the system is the sum of the harmonic spring energies:

$$
U(x_1, x_2) = \frac{1}{2}k_1\big(x_1 - x_0 - L_1\big)^2 + \frac{1}{2}k_2\big((x_2-x_1) - L_2\big)^2 + \frac{1}{2}k_3\big((x_3-x_2) - L_3\big)^2
$$

The equilibrium configuration is the $(x_1, x_2)$ that minimizes $U$. Because the total distance between the fixed ends ($12$) is larger than the sum of natural lengths ($L_1+L_2+L_3=9$), every spring is forced to stretch somewhat — so the equilibrium is a genuine trade-off between the three springs, not a trivial answer. This is exactly the kind of geometry relaxation problem that molecular mechanics force fields solve, just with more atoms and more complex potentials (Lennard-Jones, angle terms, etc.).

The gradient (force balance conditions) and Hessian (curvature / stiffness matrix) can be derived analytically:

$$
\frac{\partial U}{\partial x_1} = k_1(x_1 - x_0 - L_1) - k_2\big((x_2-x_1)-L_2\big)
$$

$$
\frac{\partial U}{\partial x_2} = k_2\big((x_2-x_1)-L_2\big) - k_3\big((x_3-x_2)-L_3\big)
$$

Since $U$ is quadratic in $x_1, x_2$, the Hessian is constant:

$$
H = \begin{pmatrix} k_1+k_2 & -k_2 \ -k_2 & k_2+k_3 \end{pmatrix}
$$

This lets us solve for the exact minimum with plain linear algebra — a perfect way to double-check our numerical optimizers.

2. Full Python Source Code

Copy this entire cell into a single Google Colaboratory cell and run it.

# ============================================================
# Two-Variable Potential Energy Minimization: Spring Chain Model
# ============================================================
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D  # noqa: F401 (enables 3D projection)
from scipy.optimize import minimize
import time

# ------------------------------------------------------------
# 1. Physical parameters
# ------------------------------------------------------------
k1, k2, k3 = 1.0, 2.0, 1.5      # spring constants
L1, L2, L3 = 3.0, 3.0, 3.0      # natural lengths
x0_fixed, x3_fixed = 0.0, 12.0  # fixed endpoints

def potential_energy(x):
    """Total potential energy U(x1, x2) of the spring chain."""
    x1, x2 = x
    U1 = 0.5 * k1 * (x1 - x0_fixed - L1) ** 2
    U2 = 0.5 * k2 * ((x2 - x1) - L2) ** 2
    U3 = 0.5 * k3 * ((x3_fixed - x2) - L3) ** 2
    return U1 + U2 + U3

def gradient(x):
    """Analytic gradient (force imbalance) of U."""
    x1, x2 = x
    dU_dx1 = k1 * (x1 - x0_fixed - L1) - k2 * ((x2 - x1) - L2)
    dU_dx2 = k2 * ((x2 - x1) - L2) - k3 * ((x3_fixed - x2) - L3)
    return np.array([dU_dx1, dU_dx2])

def hessian(x):
    """Analytic (constant) Hessian matrix of U."""
    h11 = k1 + k2
    h22 = k2 + k3
    h12 = -k2
    return np.array([[h11, h12], [h12, h22]])

# ------------------------------------------------------------
# 2. Energy landscape: SLOW (naive loop) vs FAST (vectorized)
# ------------------------------------------------------------
x1_range = np.linspace(-2, 10, 300)
x2_range = np.linspace(0, 14, 300)

# --- Naive version: nested Python loops (slow) ---
t0 = time.time()
X1, X2 = np.meshgrid(x1_range, x2_range)
U_slow = np.zeros_like(X1)
for i in range(X1.shape[0]):
    for j in range(X1.shape[1]):
        U_slow[i, j] = potential_energy([X1[i, j], X2[i, j]])
t_slow = time.time() - t0

# --- Vectorized version: full NumPy array math (fast) ---
t0 = time.time()
U1_grid = 0.5 * k1 * (X1 - x0_fixed - L1) ** 2
U2_grid = 0.5 * k2 * ((X2 - X1) - L2) ** 2
U3_grid = 0.5 * k3 * ((x3_fixed - X2) - L3) ** 2
U_fast = U1_grid + U2_grid + U3_grid
t_fast = time.time() - t0

print(f"Naive loop computation time:      {t_slow*1000:.2f} ms")
print(f"Vectorized computation time:      {t_fast*1000:.2f} ms")
print(f"Speedup factor:                   {t_slow/t_fast:.1f}x")
print(f"Results identical (max diff):     {np.max(np.abs(U_slow - U_fast)):.2e}")

# ------------------------------------------------------------
# 3. Method A: Custom steepest-descent optimizer (records path)
# ------------------------------------------------------------
def gradient_descent(x_start, lr=0.15, tol=1e-8, max_iter=500):
    path = [np.array(x_start, dtype=float)]
    x = np.array(x_start, dtype=float)
    for _ in range(max_iter):
        grad = gradient(x)
        x = x - lr * grad
        path.append(x.copy())
        if np.linalg.norm(grad) < tol:
            break
    return x, np.array(path)

x_start = np.array([1.0, 8.0])
x_gd, path_gd = gradient_descent(x_start)

# ------------------------------------------------------------
# 4. Method B: scipy.optimize.minimize with analytic gradient/Hessian
# ------------------------------------------------------------
result = minimize(
    potential_energy, x_start,
    jac=gradient, hess=hessian,
    method='Newton-CG',
    options={'xtol': 1e-10}
)
x_scipy = result.x

# ------------------------------------------------------------
# 5. Method C: Exact analytical solution (linear system H x = c)
# ------------------------------------------------------------
H = hessian(x_start)  # constant everywhere since U is quadratic
c = np.array([
    k1 * (x0_fixed + L1) - k2 * L2,
    k2 * L2 + k3 * (x3_fixed - L3)
])
x_analytic = np.linalg.solve(H, c)

# ------------------------------------------------------------
# 6. Print summary of results
# ------------------------------------------------------------
print("\n--- Equilibrium Positions ---")
print(f"Gradient Descent : x1={x_gd[0]:.6f}, x2={x_gd[1]:.6f}, U={potential_energy(x_gd):.6f}")
print(f"scipy Newton-CG   : x1={x_scipy[0]:.6f}, x2={x_scipy[1]:.6f}, U={potential_energy(x_scipy):.6f}")
print(f"Analytical (exact): x1={x_analytic[0]:.6f}, x2={x_analytic[1]:.6f}, U={potential_energy(x_analytic):.6f}")

bond1 = x_analytic[0] - x0_fixed
bond2 = x_analytic[1] - x_analytic[0]
bond3 = x3_fixed - x_analytic[1]
print("\n--- Equilibrium Bond (Spring) Lengths ---")
print(f"Spring 1: {bond1:.4f}  (natural length {L1})")
print(f"Spring 2: {bond2:.4f}  (natural length {L2})")
print(f"Spring 3: {bond3:.4f}  (natural length {L3})")

# ------------------------------------------------------------
# 7. Visualization: 3D energy surface
# ------------------------------------------------------------
fig = plt.figure(figsize=(14, 6))

ax1 = fig.add_subplot(1, 2, 1, projection='3d')
surf = ax1.plot_surface(X1, X2, U_fast, cmap='viridis', alpha=0.85,
                         linewidth=0, antialiased=True)
ax1.scatter(x_analytic[0], x_analytic[1], potential_energy(x_analytic),
            color='red', s=80, label='Equilibrium (minimum)')
ax1.set_xlabel('x1 (position of P1)')
ax1.set_ylabel('x2 (position of P2)')
ax1.set_zlabel('Potential Energy U')
ax1.set_title('3D Potential Energy Surface U(x1, x2)')
ax1.legend()
fig.colorbar(surf, ax=ax1, shrink=0.5, aspect=10, label='Energy')

# ------------------------------------------------------------
# 8. Visualization: 2D contour plot with optimization path
# ------------------------------------------------------------
ax2 = fig.add_subplot(1, 2, 2)
contour = ax2.contourf(X1, X2, U_fast, levels=40, cmap='viridis')
ax2.contour(X1, X2, U_fast, levels=15, colors='white', linewidths=0.4, alpha=0.5)
ax2.plot(path_gd[:, 0], path_gd[:, 1], 'r.-', markersize=3, linewidth=1,
         label='Gradient descent path')
ax2.plot(x_start[0], x_start[1], 'ko', markersize=8, label='Start point')
ax2.plot(x_analytic[0], x_analytic[1], 'r*', markersize=18, label='Equilibrium')
ax2.set_xlabel('x1 (position of P1)')
ax2.set_ylabel('x2 (position of P2)')
ax2.set_title('Energy Contours and Descent Path to Equilibrium')
ax2.legend()
fig.colorbar(contour, ax=ax2, label='Energy')

plt.tight_layout()
plt.show()

Naive loop computation time:      558.96 ms
Vectorized computation time:      6.64 ms
Speedup factor:                   84.2x
Results identical (max diff):     2.84e-14

--- Equilibrium Positions ---
Gradient Descent : x1=4.384615, x2=8.076923, U=2.076923
scipy Newton-CG   : x1=4.384615, x2=8.076923, U=2.076923
Analytical (exact): x1=4.384615, x2=8.076923, U=2.076923

--- Equilibrium Bond (Spring) Lengths ---
Spring 1: 4.3846  (natural length 3.0)
Spring 2: 3.6923  (natural length 3.0)
Spring 3: 3.9231  (natural length 3.0)

3. Code Walkthrough

Physical model functions

potential_energy(x) implements the equation for $U(x_1,x_2)$ directly — three harmonic terms, one per spring. gradient(x) and hessian(x) implement the analytic derivatives we derived above. Supplying these analytically (rather than letting the optimizer approximate them with finite differences) is both faster and numerically more accurate, since finite-difference gradients require multiple extra function evaluations and introduce rounding error.

Naive loop vs. vectorized grid computation

This is the “time-consuming part” of the problem, and it’s a great illustration of why vectorization matters in scientific Python. Computing the energy value at every point of a 300×300 grid via a for i in range(...): for j in range(...): double loop calls potential_energy() 90,000 times in pure Python, which is slow because every call carries Python’s interpreter overhead. The vectorized version instead evaluates the exact same formula on entire NumPy arrays at once (X1, X2 are full 300×300 arrays), letting NumPy’s compiled C backend do all 90,000 evaluations in one shot. The script prints both timings and the speedup factor — on a typical Colab CPU you should see roughly a 50–150× speedup, and the “max diff” check confirms both methods produce identical results (up to floating-point rounding).

Three independent solving methods

Gradient descent (gradient_descent) — a hand-written steepest-descent loop that repeatedly steps opposite to the gradient. We record every intermediate point in path_gd so we can visualize how the optimizer walks downhill.
scipy’s minimize with method='Newton-CG' — a professional-grade optimizer that uses our analytic gradient and Hessian to converge in far fewer iterations than plain gradient descent, since it accounts for the curvature of the energy surface.
Exact analytical solution — because $U$ is a quadratic function, its minimum satisfies the linear equation $H\mathbf{x} = \mathbf{c}$, which np.linalg.solve solves exactly in one step. This serves as a ground-truth check: both numerical methods should match it almost to machine precision.

All three results are printed together, along with the resulting equilibrium bond lengths (the actual stretched length of each spring), so you can see how each spring deviates from its natural length to balance the system.

Visualization

The 3D surface plot (left panel) shows the full bowl-shaped energy landscape — since our potential is a sum of quadratics, it’s a paraboloid, and the red dot marks the single global minimum. The contour plot with path overlay (right panel) shows the same landscape from above, with the gradient descent trajectory drawn as a red line from the black starting point down to the red star at equilibrium. Watching the descent path curve toward the minimum (rather than moving in a straight line) illustrates that the two variables are coupled — moving $x_1$ affects the optimal $x_2$ and vice versa, exactly as spring 2 couples the two free particles together.

4. Interpreting the Results

Once you run the code, check that:

All three methods agree — gradient descent, scipy’s Newton-CG, and the exact linear solve should all report essentially the same $(x_1, x_2)$ and energy value (differences should be smaller than $10^{-4}$).
The bond lengths make physical sense — the stiffer spring ($k_2=2.0$) should be stretched less relative to its natural length than the softer springs, since it “resists” deformation more strongly. This is a direct numerical demonstration of Hooke’s law competition between coupled springs.
The energy surface is convex (bowl-shaped) — this is why gradient descent, despite being a simple algorithm, reliably finds the global minimum here. Real molecular potentials (e.g., Lennard-Jones) are not globally convex and can have multiple local minima, which is why more sophisticated global optimization or multiple random restarts are used in real molecular mechanics software.

5. Extending the Model

This two-variable spring chain is a minimal but genuine example of geometry optimization, the same core computational idea used in molecular simulation packages (like force-field minimization in GROMACS or AMBER) to relax a molecule into its stable 3D shape. Natural extensions worth trying:

Replace one harmonic spring with a Lennard-Jones potential $U_{LJ}(r) = 4\epsilon\left[(\sigma/r)^{12}-(\sigma/r)^6\right]$ to introduce a non-convex landscape with a local minimum, and see how gradient descent can get stuck depending on the starting point.
Extend to more free particles (higher dimensions), where visualizing the full energy surface is no longer possible, but the same gradient/Hessian-based optimization strategy still works.
Add a 2D or 3D geometry (particles free to move in the plane, not just along a line) to more closely resemble real bond-angle relaxation in molecules.

Minimizing the Goldstein-Price Function with Python

August 8, 2026

A Global Optimization Walkthrough

Global optimization is one of those problems that looks simple on paper but gets nasty fast once your objective function has multiple local minima. A classic benchmark for testing optimization algorithms is the Goldstein-Price function, a two-dimensional function riddled with local minima that only reveals its single global minimum after careful searching.

In this article, we’ll define the function, minimize it with Python and SciPy, visualize the results in 3D, and break down every part of the code so you understand exactly what’s happening under the hood.

What Is the Goldstein-Price Function?

The Goldstein-Price function is defined as:

$$
f(x, y) = \left[1 + (x + y + 1)^2 \left(19 - 14x + 3x^2 - 14y + 6xy + 3y^2\right)\right] \times \left[30 + (2x - 3y)^2 \left(18 - 32x + 12x^2 + 48y - 36xy + 27y^2\right)\right]
$$

It is typically evaluated over the domain:

$$
x, y \in [-2, 2]
$$

The function has a known global minimum of:

$$
f(0, -1) = 3
$$

What makes this function a good stress test is that it’s not smooth and simple — it has several local minima and a huge dynamic range (values swing from 3 up to over a million within the search domain), which means naive gradient-based methods starting from a bad initial guess can easily get stuck.

To reliably find the global minimum, we’ll use a two-step approach:

Differential Evolution — a population-based global optimization algorithm from SciPy that doesn’t require gradients and is good at escaping local minima.
Nelder-Mead — a local simplex-based method, used here to confirm/refine a result from a specific starting point, illustrating how a local method can get “close enough” but benefits from a good starting guess.

The Source Code

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from scipy.optimize import minimize, differential_evolution
import time

# --- Define the Goldstein-Price function (scalar version, for optimizers) ---
def goldstein_price(v):
    x, y = v
    term1 = 1 + (x + y + 1)**2 * (19 - 14*x + 3*x**2 - 14*y + 6*x*y + 3*y**2)
    term2 = 30 + (2*x - 3*y)**2 * (18 - 32*x + 12*x**2 + 48*y - 36*x*y + 27*y**2)
    return term1 * term2

# --- Vectorized version (for fast grid evaluation used in plotting) ---
def goldstein_price_vec(X, Y):
    term1 = 1 + (X + Y + 1)**2 * (19 - 14*X + 3*X**2 - 14*Y + 6*X*Y + 3*Y**2)
    term2 = 30 + (2*X - 3*Y)**2 * (18 - 32*X + 12*X**2 + 48*Y - 36*X*Y + 27*Y**2)
    return term1 * term2

# --- Search domain ---
bounds = [(-2, 2), (-2, 2)]

# --- Step 1: Global search using Differential Evolution ---
start_time = time.time()
result_de = differential_evolution(
    goldstein_price,
    bounds,
    tol=1e-12,
    seed=42
)
elapsed = time.time() - start_time

# --- Step 2: Local refinement from an arbitrary starting point ---
result_local = minimize(
    goldstein_price,
    x0=[0.5, -0.5],
    method='Nelder-Mead'
)

# --- Report results ---
print("=== Differential Evolution (Global Search) ===")
print(f"  x = {result_de.x[0]:.6f}, y = {result_de.x[1]:.6f}")
print(f"  f(x, y) = {result_de.fun:.6f}")
print(f"  Elapsed time: {elapsed:.4f} sec")

print("\n=== Nelder-Mead (Local Search from [0.5, -0.5]) ===")
print(f"  x = {result_local.x[0]:.6f}, y = {result_local.x[1]:.6f}")
print(f"  f(x, y) = {result_local.fun:.6f}")

=== Differential Evolution (Global Search) ===
  x = -0.000000, y = -1.000000
  f(x, y) = 3.000000
  Elapsed time: 0.2694 sec

=== Nelder-Mead (Local Search from [0.5, -0.5]) ===
  x = 0.000019, y = -0.999989
  f(x, y) = 3.000000

Code Walkthrough

The two function definitions. goldstein_price(v) takes a single vector v = [x, y] and returns a scalar — this is the signature SciPy’s optimizers expect. goldstein_price_vec(X, Y) does the exact same math but accepts NumPy arrays (meshgrids) directly, using array broadcasting instead of loops. This second version is purely for speed when we evaluate the function over thousands of grid points for plotting — looping point-by-point in Python would be dramatically slower, so vectorizing with NumPy’s broadcasted arithmetic keeps the whole grid evaluation to a fraction of a second.

differential_evolution is a stochastic, population-based global optimizer. Instead of following a gradient, it maintains a population of candidate solutions and evolves them generation by generation using mutation and crossover, which makes it well-suited to functions like Goldstein-Price that have multiple local minima capable of trapping gradient-based methods. We pass tol=1e-12 to tighten the convergence criterion and seed=42 to make the result reproducible.

minimize(..., method='Nelder-Mead') performs a local simplex search starting from [0.5, -0.5]. Because Nelder-Mead doesn’t require derivatives, it works even on non-smooth objective functions, but it only guarantees convergence to whichever minimum is closest to the starting point — not necessarily the global one. Comparing its output to the differential evolution result is a good way to see how starting-point-dependent local optimization can be.

When you run this, both methods should converge to essentially the same point, (x, y) ≈ (0, -1) with f(x, y) ≈ 3, which matches the function’s known global minimum. The differential evolution run also typically finishes in well under a second, since the search domain is small and two-dimensional.

Visualizing the Function

Because Goldstein-Price spans values from 3 to over a million across the domain, plotting it on a linear scale would flatten the interesting structure near the minimum into an indistinguishable blob. The fix is to plot the base-10 logarithm of the function instead, which compresses the huge dynamic range into something visually readable while preserving the location of the minimum.

# --- Build a grid over the search domain ---
x = np.linspace(-2, 2, 200)
y = np.linspace(-2, 2, 200)
X, Y = np.meshgrid(x, y)
Z = goldstein_price_vec(X, Y)
Z_log = np.log10(Z)  # compress the huge dynamic range for visualization

fig = plt.figure(figsize=(16, 6))

# --- 3D surface plot (log scale) ---
ax1 = fig.add_subplot(1, 2, 1, projection='3d')
surf = ax1.plot_surface(X, Y, Z_log, cmap='viridis', alpha=0.9, edgecolor='none')
ax1.set_xlabel('x')
ax1.set_ylabel('y')
ax1.set_zlabel('log10(f(x, y))')
ax1.set_title('Goldstein-Price Function (log scale)')
ax1.scatter(
    [result_de.x[0]], [result_de.x[1]], [np.log10(result_de.fun)],
    color='red', s=100, marker='*', label='Global Minimum'
)
ax1.legend()
fig.colorbar(surf, ax=ax1, shrink=0.5)

# --- 2D contour plot (log scale) ---
ax2 = fig.add_subplot(1, 2, 2)
contour = ax2.contourf(X, Y, Z_log, levels=50, cmap='viridis')
ax2.contour(X, Y, Z_log, levels=20, colors='white', linewidths=0.3, alpha=0.5)
ax2.plot(result_de.x[0], result_de.x[1], 'r*', markersize=20, label='Global Minimum')
ax2.set_xlabel('x')
ax2.set_ylabel('y')
ax2.set_title('Contour Plot (log scale)')
ax2.legend()
fig.colorbar(contour, ax=ax2)

plt.tight_layout()
plt.show()

Graph Explanation

Left panel — 3D surface plot. This shows the shape of log10(f(x, y)) across the search domain. You can clearly see a steep, funnel-shaped basin near (0, -1), marked with a red star, which is the global minimum. Away from that basin, the surface rises sharply into several ridges and bumps — these are the local minima and saddle regions that make this function a challenging optimization benchmark. The logarithmic z-axis is what makes both the deep basin and the surrounding terrain visible in the same plot; without it, the basin would be invisible next to the much larger values elsewhere in the domain.

Right panel — 2D contour plot. This is essentially a bird’s-eye view of the same log-scaled surface, where color represents function value (darker purple = lower, i.e. closer to the minimum) and the white contour lines trace constant-value bands. The red star again marks the global minimum at (0, -1). Notice how tightly the contour lines bunch up around that point — that steepness is exactly why gradient-based optimizers can converge quickly once they’re near it, but can also get misled by the other local minima elsewhere on the map if they start too far away.

Together, these two plots make it easy to visually confirm what the numerical optimizers already told us: the true minimum sits at (0, -1) with a function value of 3, nestled at the bottom of a narrow, steep-sided basin surrounded by a much bumpier landscape.

Taming the Beale Function

August 7, 2026

A Journey Through a Deceptively Simple Optimization Landscape

Introduction

Among the classic benchmark functions used to test optimization algorithms, the Beale function holds a special place. It looks innocent enough on paper — just a sum of three squared terms — but its landscape hides sharp, narrow valleys that can trap naive gradient-based solvers. In this article, we’ll dissect the Beale function mathematically, implement a robust two-stage optimization pipeline in Python, and visualize the entire search process in 3D.

The Mathematics of the Beale Function

The Beale function is defined as:

$$
f(x, y) = (1.5 - x + xy)^2 + (2.25 - x + xy^2)^2 + (2.625 - x + xy^3)^2
$$

It is evaluated over the domain $x, y \in [-4.5, 4.5]$, and its global minimum is known analytically:

$$
f(3, ; 0.5) = 0
$$

What makes this function tricky is the steep, curved valley leading toward the minimum, combined with flat plateaus and extremely large function values near the corners of the domain (values can exceed $10^5$). This asymmetry means a single fixed-step gradient method can easily overshoot or stall, which is exactly why a hybrid global-then-local strategy is the right tool for the job.

Optimization Strategy

Rather than relying purely on a local method (which is highly sensitive to the starting point) or purely on a slow global method, we combine two techniques:

Differential Evolution (DE) — a population-based global optimizer that explores the whole domain without needing gradient information, giving us a good approximate basin.
L-BFGS-B — a fast, gradient-based local refinement step that polishes the DE result down to near machine precision.

This two-stage approach is both fast and reliable: DE avoids getting stuck in the wrong region, while L-BFGS-B converges to high accuracy in a handful of iterations once we’re already close.

Source Code

import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import differential_evolution, minimize
import time

plt.style.use('dark_background')

# -----------------------------
# Beale function definition
# -----------------------------
def beale(params):
    x, y = params
    term1 = (1.5 - x + x * y) ** 2
    term2 = (2.25 - x + x * y ** 2) ** 2
    term3 = (2.625 - x + x * y ** 3) ** 2
    return term1 + term2 + term3

# -----------------------------
# History tracking for visualization
# -----------------------------
convergence_history = []
positions_history = []

def de_callback(xk, convergence):
    positions_history.append(np.array(xk, dtype=float))
    convergence_history.append(beale(xk))

bounds = [(-4.5, 4.5), (-4.5, 4.5)]

# -----------------------------
# Stage 1: Global search with Differential Evolution
# -----------------------------
start_time = time.time()
result_de = differential_evolution(
    beale,
    bounds,
    seed=42,
    maxiter=300,
    tol=1e-12,
    mutation=(0.5, 1.5),
    recombination=0.7,
    callback=de_callback,
    polish=False
)
de_elapsed = time.time() - start_time

# -----------------------------
# Stage 2: Local refinement with L-BFGS-B
# -----------------------------
start_time = time.time()
result_local = minimize(
    beale,
    result_de.x,
    method='L-BFGS-B',
    bounds=bounds,
    tol=1e-15
)
local_elapsed = time.time() - start_time

true_min = np.array([3.0, 0.5])
distance_to_true = np.linalg.norm(result_local.x - true_min)

print("=" * 60)
print("Beale Function Minimization Results")
print("=" * 60)
print(f"[Stage 1] Differential Evolution")
print(f"  x = {result_de.x[0]:.10f}")
print(f"  y = {result_de.x[1]:.10f}")
print(f"  f(x,y) = {result_de.fun:.15f}")
print(f"  Generations: {result_de.nit}, Time: {de_elapsed:.4f}s")
print("-" * 60)
print(f"[Stage 2] L-BFGS-B Local Refinement")
print(f"  x = {result_local.x[0]:.10f}")
print(f"  y = {result_local.x[1]:.10f}")
print(f"  f(x,y) = {result_local.fun:.2e}")
print(f"  Time: {local_elapsed:.4f}s")
print("-" * 60)
print(f"Known global minimum: (3.0, 0.5), f = 0.0")
print(f"Distance to true minimum: {distance_to_true:.2e}")
print(f"Total optimization time: {de_elapsed + local_elapsed:.4f}s")
print("=" * 60)

============================================================
Beale Function Minimization Results
============================================================
[Stage 1] Differential Evolution
  x = 3.0000000000
  y = 0.5000000000
  f(x,y) = 0.000000000000000
  Generations: 146, Time: 0.6139s
------------------------------------------------------------
[Stage 2] L-BFGS-B Local Refinement
  x = 3.0000000000
  y = 0.5000000000
  f(x,y) = 3.20e-31
  Time: 0.0116s
------------------------------------------------------------
Known global minimum: (3.0, 0.5), f = 0.0
Distance to true minimum: 4.97e-16
Total optimization time: 0.6255s
============================================================

Code Walkthrough

The objective function. beale() implements the three-term formula exactly as written mathematically. Because scipy.optimize passes parameters as a single array, we unpack x, y = params at the top of the function.

History tracking. The de_callback function is invoked after every generation of the differential evolution algorithm. We store both the candidate position (positions_history) and its function value (convergence_history), which lets us later draw the search trajectory over the contour map and plot the convergence curve.

Stage 1 — global search. differential_evolution maintains a population of candidate solutions and evolves them via mutation and crossover, requiring no gradient information. We disable the built-in polish step (polish=False) because we handle refinement ourselves in Stage 2, giving us explicit control and separate timing for each stage.

Stage 2 — local refinement. L-BFGS-B takes the best point found by DE and rapidly converges toward the true minimum using quasi-Newton updates with box constraints, typically needing only a few iterations since the starting point is already close to the optimum.

Why this is already fast. For a 2D problem like this, DE with 300 generations and a small population converges in well under a second, and L-BFGS-B refinement adds negligible overhead. No further acceleration (e.g., multiprocessing or vectorized batch evaluation) is necessary here — the bottleneck for 2D benchmark functions is never raw computation time.

Visualizing the Optimization Landscape

Numbers alone don’t do justice to how treacherous the Beale function’s landscape really is. Let’s render it in three complementary views: a 3D surface, a contour map with the search trajectory overlaid, and the convergence curve.

from matplotlib.gridspec import GridSpec
from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure(figsize=(16, 12))
fig.patch.set_facecolor('#0d1117')
gs = GridSpec(2, 2, width_ratios=[1.3, 1], height_ratios=[1, 1],
              hspace=0.35, wspace=0.3, figure=fig)

# -----------------------------
# Prepare grid data
# -----------------------------
x_range = np.linspace(-4.5, 4.5, 220)
y_range = np.linspace(-4.5, 4.5, 220)
X, Y = np.meshgrid(x_range, y_range)
Z = (1.5 - X + X * Y) ** 2 + (2.25 - X + X * Y ** 2) ** 2 + (2.625 - X + X * Y ** 3) ** 2
Z_log = np.log1p(Z)  # log-scale to compress the huge dynamic range

# -----------------------------
# Panel 1: 3D Surface
# -----------------------------
ax1 = fig.add_subplot(gs[:, 0], projection='3d')
ax1.set_facecolor('#0d1117')
surf = ax1.plot_surface(X, Y, Z_log, cmap='inferno', edgecolor='none',
                         alpha=0.92, antialiased=True, rcount=150, ccount=150)
ax1.scatter([3.0], [0.5], [np.log1p(beale([3.0, 0.5]))],
            color='cyan', s=140, marker='*', edgecolor='white',
            linewidth=0.5, label='Global Minimum (3, 0.5)', depthshade=False)
ax1.scatter([result_local.x[0]], [result_local.x[1]], [np.log1p(result_local.fun)],
            color='lime', s=80, marker='o', edgecolor='white',
            linewidth=0.5, label='Found Minimum', depthshade=False)
ax1.set_xlabel('x', color='white', labelpad=10)
ax1.set_ylabel('y', color='white', labelpad=10)
ax1.set_zlabel('log(1 + f(x,y))', color='white', labelpad=10)
ax1.set_title('Beale Function Surface (log scale)', color='white', fontsize=13, pad=15)
ax1.tick_params(colors='white')
ax1.legend(loc='upper left', facecolor='#161b22', edgecolor='gray', labelcolor='white', fontsize=9)
ax1.view_init(elev=28, azim=-55)
fig.colorbar(surf, ax=ax1, shrink=0.5, pad=0.08)

# -----------------------------
# Panel 2: Contour + search trace
# -----------------------------
ax2 = fig.add_subplot(gs[0, 1])
ax2.set_facecolor('#0d1117')
ax2.contourf(X, Y, Z_log, levels=50, cmap='inferno')
positions_arr = np.array(positions_history)
if len(positions_arr) > 0:
    ax2.plot(positions_arr[:, 0], positions_arr[:, 1],
              color='cyan', linewidth=1.2, alpha=0.8, label='DE search trace')
ax2.scatter([3.0], [0.5], color='lime', s=160, marker='*',
            edgecolor='white', linewidth=0.5, label='Global Minimum', zorder=5)
ax2.set_xlabel('x', color='white')
ax2.set_ylabel('y', color='white')
ax2.set_title('Contour Map with Search Path', color='white', fontsize=13)
ax2.tick_params(colors='white')
ax2.legend(facecolor='#161b22', edgecolor='gray', labelcolor='white', fontsize=8, loc='upper left')

# -----------------------------
# Panel 3: Convergence curve
# -----------------------------
ax3 = fig.add_subplot(gs[1, 1])
ax3.set_facecolor('#0d1117')
conv_arr = np.array(convergence_history) + 1e-16
ax3.semilogy(range(len(conv_arr)), conv_arr, color='#ff6b6b', linewidth=2)
ax3.set_xlabel('Generation', color='white')
ax3.set_ylabel('f(x, y)  [log scale]', color='white')
ax3.set_title('Differential Evolution Convergence', color='white', fontsize=13)
ax3.grid(True, alpha=0.2)
ax3.tick_params(colors='white')

plt.tight_layout()
plt.show()

Interpreting the Visualization

The 3D surface plot (left panel) reveals why this function is such a good stress test: the terrain is dominated by a steep-walled basin curling from the upper-left toward the bottom-right, with the true minimum sitting at the bottom of a long, narrow trough. We plot $\log(1+f)$ instead of raw $f$ because the true function values span from $0$ to over $10^5$ across the domain — without the log transform, the entire interesting region near the minimum would be crushed flat and invisible.

The contour map with search trace (top right) shows the differential evolution population converging generation by generation. Notice how the cyan trace initially explores broadly across the domain before narrowing sharply into the valley containing the star-marked global minimum — this is the hallmark of a well-functioning global optimizer.

The convergence curve (bottom right) plots the best function value found at each generation on a log scale. The steep initial drop reflects DE quickly identifying the correct basin, while the long, near-flat tail shows the algorithm fine-tuning within that basin before we hand off to L-BFGS-B for the final high-precision polish.

Conclusion

The Beale function is a textbook example of why optimization algorithm choice matters as much as the algorithm’s raw speed. A purely local method risks getting misled by the function’s sharp curvature and vast scale differences, while a purely global method wastes time achieving precision it isn’t designed for. By pairing differential evolution’s broad exploration with L-BFGS-B’s precise convergence, we consistently land within machine-precision distance of the true minimum at $(3, 0.5)$ — a pattern that generalizes well beyond this single benchmark function to many real-world non-convex optimization problems.

Minimizing the Sphere Function

August 6, 2026

A Clean Introduction to Convex Optimization

Introduction

Among all the benchmark functions used in numerical optimization, the Sphere function stands out for its simplicity. It is smooth, strictly convex, and has a single global minimum — making it the perfect starting point for understanding how optimization algorithms behave before tackling more complex, non-convex landscapes. In this article, we implement and visualize the minimization of the Sphere function in two dimensions, comparing a hand-written gradient descent routine against SciPy’s BFGS optimizer.

Mathematical Definition

The Sphere function in $n$ dimensions is defined as:

$$
f(\mathbf{x}) = \sum_{i=1}^{n} x_i^2
$$

For our two-dimensional example, this simplifies to:

$$
f(x_1, x_2) = x_1^2 + x_2^2
$$

Because the function is a simple sum of squares, its gradient has a closed-form expression:

$$
\nabla f(\mathbf{x}) = 2\mathbf{x}
$$

and its Hessian matrix is constant and positive definite:

$$
H(\mathbf{x}) = 2I
$$

where $I$ is the identity matrix. This positive-definite Hessian is exactly what makes the Sphere function strictly convex, guaranteeing that any local minimum found is also the global minimum, located at $\mathbf{x}^* = \mathbf{0}$ with $f(\mathbf{x}^*) = 0$.

Python Implementation

The code below defines the Sphere function and its gradient, runs a custom gradient descent optimizer, cross-validates the result with SciPy’s BFGS method, and visualizes both optimization paths on a 3D surface, a contour map, and a convergence plot.

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from scipy.optimize import minimize

# Dark theme for all plots
plt.style.use('dark_background')

# ----------------------------------------------------------------------
# 1. Define the Sphere function and its gradient
# ----------------------------------------------------------------------
def sphere(x):
    """Sphere function: f(x) = sum(x_i^2)"""
    x = np.asarray(x, dtype=float)
    return np.sum(x ** 2)

def sphere_gradient(x):
    """Gradient of the Sphere function: grad f(x) = 2x"""
    x = np.asarray(x, dtype=float)
    return 2.0 * x

# ----------------------------------------------------------------------
# 2. Custom gradient descent implementation
# ----------------------------------------------------------------------
def gradient_descent(f, grad, x0, learning_rate=0.25, max_iter=60, tol=1e-10):
    x = np.array(x0, dtype=float)
    path = [x.copy()]
    values = [f(x)]

    for _ in range(max_iter):
        g = grad(x)
        x = x - learning_rate * g
        path.append(x.copy())
        values.append(f(x))
        if np.linalg.norm(g) < tol:
            break

    return np.array(path), np.array(values)

# Starting point, deliberately far from the minimum
x0 = np.array([4.0, 4.5])

gd_path, gd_values = gradient_descent(sphere, sphere_gradient, x0,
                                       learning_rate=0.25, max_iter=60)

# ----------------------------------------------------------------------
# 3. Reference solution using SciPy's BFGS optimizer
# ----------------------------------------------------------------------
bfgs_path = [x0.copy()]

def record_step(xk):
    bfgs_path.append(np.array(xk, dtype=float))

result = minimize(sphere, x0, jac=sphere_gradient, method='BFGS',
                   callback=record_step, tol=1e-10)

bfgs_path = np.array(bfgs_path)
bfgs_values = np.array([sphere(p) for p in bfgs_path])

# ----------------------------------------------------------------------
# 4. Print a short numerical summary
# ----------------------------------------------------------------------
print("=" * 60)
print("Sphere Function Minimization Summary")
print("=" * 60)
print(f"Starting point            : {x0}")
print(f"Gradient Descent result   : {gd_path[-1]}, f = {gd_values[-1]:.3e}, iterations = {len(gd_path)-1}")
print(f"BFGS result                : {result.x}, f = {result.fun:.3e}, iterations = {result.nit}")
print(f"True global minimum        : [0. 0.], f = 0.0")
print("=" * 60)

# ----------------------------------------------------------------------
# 5. Visualization
# ----------------------------------------------------------------------
fig = plt.figure(figsize=(20, 6))
fig.patch.set_facecolor('#0d0d0d')

grid_range = np.linspace(-5, 5, 150)
X, Y = np.meshgrid(grid_range, grid_range)
Z = X ** 2 + Y ** 2

# --- 5.1  3D surface with the gradient descent path ---
ax1 = fig.add_subplot(1, 3, 1, projection='3d')
ax1.set_facecolor('#0d0d0d')
surf = ax1.plot_surface(X, Y, Z, cmap='plasma', alpha=0.55,
                         edgecolor='none', antialiased=True)
ax1.plot(gd_path[:, 0], gd_path[:, 1], gd_values,
          color='#00ffea', marker='o', markersize=3, linewidth=2,
          label='Gradient Descent path')
ax1.scatter([0], [0], [0], color='white', s=60, marker='*',
            label='Global minimum')
ax1.set_xlabel('x1')
ax1.set_ylabel('x2')
ax1.set_zlabel('f(x1, x2)')
ax1.set_title('Sphere Function Surface & Descent Path', color='white')
ax1.legend(loc='upper left', fontsize=8)
fig.colorbar(surf, ax=ax1, shrink=0.5, pad=0.1)

# --- 5.2  Contour plot comparing both optimizers ---
ax2 = fig.add_subplot(1, 3, 2)
ax2.set_facecolor('#0d0d0d')
contour = ax2.contour(X, Y, Z, levels=25, cmap='plasma')
ax2.clabel(contour, inline=True, fontsize=7)
ax2.plot(gd_path[:, 0], gd_path[:, 1], color='#00ffea',
         marker='o', markersize=3, linewidth=1.5, label='Gradient Descent')
ax2.plot(bfgs_path[:, 0], bfgs_path[:, 1], color='#ff5df1',
         marker='s', markersize=4, linewidth=1.5, label='BFGS')
ax2.scatter([0], [0], color='white', s=80, marker='*', zorder=5,
            label='Global minimum')
ax2.set_xlabel('x1')
ax2.set_ylabel('x2')
ax2.set_title('Optimization Paths (Contour View)', color='white')
ax2.legend(fontsize=8)

# --- 5.3  Convergence curve (log scale) ---
ax3 = fig.add_subplot(1, 3, 3)
ax3.set_facecolor('#0d0d0d')
ax3.plot(gd_values, color='#00ffea', marker='o', markersize=3,
         label='Gradient Descent')
ax3.plot(bfgs_values, color='#ff5df1', marker='s', markersize=4,
         label='BFGS')
ax3.set_yscale('log')
ax3.set_xlabel('Iteration')
ax3.set_ylabel('f(x)  (log scale)')
ax3.set_title('Convergence Comparison', color='white')
ax3.legend(fontsize=8)
ax3.grid(True, alpha=0.3)

plt.tight_layout()
plt.show()

Code Explanation

Function and gradient definitions. sphere() computes $\sum x_i^2$ using np.sum(x ** 2), which works for any dimensionality. sphere_gradient() returns $2\mathbf{x}$ directly, avoiding the need for numerical differentiation and giving both optimizers an exact, noise-free gradient signal.

Custom gradient descent. The gradient_descent() function implements the simplest possible first-order optimizer: at each step it moves in the direction opposite to the gradient, scaled by a fixed learning_rate. Because the Sphere function’s Hessian is $2I$, the update rule simplifies to $x_{k+1} = x_k(1 - 2\eta)$, where $\eta$ is the learning rate. With $\eta = 0.25$, this factor equals $0.5$, so the distance to the origin is halved at every iteration — a textbook example of linear convergence. The loop also includes an early-stopping condition based on gradient norm, so it won’t run needless iterations once it’s essentially converged.

BFGS reference. SciPy’s minimize() with method='BFGS' is a quasi-Newton method that builds an approximate Hessian from gradient information as it iterates. Since the true Hessian of the Sphere function is a constant multiple of the identity, BFGS converges extremely fast — typically within just a handful of steps. The callback argument lets us record every intermediate point xk, which we later plot alongside the gradient descent path.

Why no separate “fast” version is needed. Because the Sphere function and its gradient are trivially cheap to evaluate (just squaring and summing a handful of numbers), this problem never becomes a computational bottleneck even at high iteration counts. The vectorized NumPy operations already run in microseconds per step, so no further speed optimization is necessary here — this keeps the code simple and readable.

Visualization and Interpretation

The figure produced by the script above contains three panels side by side:

3D Surface Plot (left). This shows the paraboloid shape of the Sphere function, colored with the plasma colormap. The cyan trajectory traces the gradient descent path as it spirals down toward the bowl’s bottom, visually confirming the smooth, funnel-like geometry that makes this function so easy to optimize.
Contour Plot (center). Viewed from directly above, the concentric circles represent level sets of equal function value. Both optimization paths are overlaid: the cyan markers show gradient descent taking small, steady steps, while the magenta squares show BFGS reaching the center in dramatically fewer steps thanks to its curvature-aware search direction.
Convergence Plot (right). Plotted on a logarithmic y-axis, this panel makes the difference in convergence speed unmistakable. Gradient descent’s function value decreases in a straight line on the log scale (confirming the theoretical linear/geometric convergence rate), while BFGS’s curve drops almost vertically, reaching machine-precision accuracy in only a few iterations.

The console output — showing the starting point, final coordinates, function values, and iteration counts for both methods — should also confirm that both optimizers converge to the same global minimum at $[0, 0]$ with $f(\mathbf{x}) \approx 0$.

============================================================
Sphere Function Minimization Summary
============================================================
Starting point            : [4.  4.5]
Gradient Descent result   : [1.45519152e-11 1.63709046e-11], f = 4.798e-22, iterations = 38
BFGS result                : [-6.66133815e-16  0.00000000e+00], f = 4.437e-31, iterations = 3
True global minimum        : [0. 0.], f = 0.0
============================================================

Conclusion

The Sphere function may be the “hello world” of optimization problems, but it offers genuine insight: it isolates the raw behavior of an algorithm’s convergence rate without the confounding effects of non-convexity, saddle points, or multiple local minima. By comparing a from-scratch gradient descent implementation against SciPy’s BFGS, we get a clear, visual demonstration of why second-order (curvature-aware) methods so dramatically outperform first-order methods on well-conditioned convex problems — a lesson that carries directly into far more complex optimization landscapes.

Minimizing Himmelblau's Function

August 5, 2026

A Journey Through Multiple Global Minima

Introduction

Optimization problems don’t always have a single “best” answer. Sometimes, a function has multiple points that are all equally optimal — and this is exactly the fascinating property of Himmelblau’s function, a classic benchmark in numerical optimization.

In this post, we’ll explore Himmelblau’s function, find all four of its global minima using Python, and visualize the results with a stunning 3D surface plot.

What is Himmelblau’s Function?

Himmelblau’s function is defined as:

$$
f(x, y) = (x^2 + y - 11)^2 + (x + y^2 - 7)^2
$$

This function is widely used to test optimization algorithms because it has four identical global minima, all with a function value of exactly 0. The minima are located at approximately:

$$
(3.0, 2.0), \quad (-2.805118, 3.131312), \quad (-3.779310, -3.283186), \quad (3.584428, -1.848126)
$$

Because there are multiple equally good solutions, a naive gradient-descent-style optimizer starting from a single point will only find one of these minima — the one closest to its starting position. To find all four, we need to try multiple starting points across the search space.

Strategy

Our approach:

Define Himmelblau’s function in Python.
Use scipy.optimize.minimize with a multi-start strategy — launching the optimizer from many different initial points across the domain.
Cluster the resulting solutions to identify the distinct global minima (removing duplicates found from different starting points).
Visualize the function as a 3D surface and mark the discovered minima.
Also show a 2D contour plot for a clearer top-down view.

Since running the optimizer from a single point is fast, but running it from hundreds of starting points could be slow if done naively, we vectorize the initial point generation with NumPy and use SciPy’s efficient BFGS-based solver (L-BFGS-B) for speed, while keeping the total number of starts modest (a grid of 100 points) so it finishes almost instantly.

Full Source Code

# ==========================================================
# Himmelblau's Function: Multi-Start Optimization + 3D Plot
# ==========================================================
import numpy as np
from scipy.optimize import minimize
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D  # noqa: F401 (enables 3D projection)

# ----------------------------------------------------------
# 1. Define Himmelblau's function and its gradient
# ----------------------------------------------------------
def himmelblau(v):
    x, y = v
    return (x**2 + y - 11)**2 + (x + y**2 - 7)**2

def himmelblau_grad(v):
    x, y = v
    dfdx = 4*x*(x**2 + y - 11) + 2*(x + y**2 - 7)
    dfdy = 2*(x**2 + y - 11) + 4*y*(x + y**2 - 7)
    return np.array([dfdx, dfdy])

# ----------------------------------------------------------
# 2. Multi-start optimization to find ALL global minima
# ----------------------------------------------------------
# Generate a grid of starting points covering the search domain
grid_n = 10  # 10x10 = 100 starting points
xs = np.linspace(-6, 6, grid_n)
ys = np.linspace(-6, 6, grid_n)
starts = np.array([[x, y] for x in xs for y in ys])

found_minima = []
tolerance = 1e-4  # distance threshold to consider two minima "the same"

for start in starts:
    result = minimize(
        himmelblau,
        start,
        jac=himmelblau_grad,
        method='L-BFGS-B',
        bounds=[(-6, 6), (-6, 6)]
    )
    if result.success and result.fun < 1e-6:  # only keep true global minima
        point = result.x
        # Check if this point is already in our list (avoid duplicates)
        is_new = True
        for existing in found_minima:
            if np.linalg.norm(point - existing) < tolerance:
                is_new = False
                break
        if is_new:
            found_minima.append(point)

found_minima = np.array(found_minima)

print(f"Number of distinct global minima found: {len(found_minima)}")
print("Coordinates of global minima:")
for i, m in enumerate(found_minima):
    print(f"  Minimum {i+1}: x = {m[0]:.6f}, y = {m[1]:.6f}, f(x,y) = {himmelblau(m):.8f}")

# ----------------------------------------------------------
# 3. Prepare data for 3D surface plot
# ----------------------------------------------------------
X = np.linspace(-6, 6, 200)
Y = np.linspace(-6, 6, 200)
X, Y = np.meshgrid(X, Y)
Z = (X**2 + Y - 11)**2 + (X + Y**2 - 7)**2

# ----------------------------------------------------------
# 4. Plot: 3D surface + 2D contour with minima marked
# ----------------------------------------------------------
fig = plt.figure(figsize=(16, 7))

# --- 3D Surface Plot ---
ax1 = fig.add_subplot(1, 2, 1, projection='3d')
surf = ax1.plot_surface(X, Y, Z, cmap='viridis', alpha=0.85, linewidth=0, antialiased=True)
ax1.scatter(
    found_minima[:, 0], found_minima[:, 1],
    [himmelblau(m) for m in found_minima],
    color='red', s=80, marker='o', label='Global minima', depthshade=False
)
ax1.set_xlabel('x')
ax1.set_ylabel('y')
ax1.set_zlabel('f(x, y)')
ax1.set_title("Himmelblau's Function - 3D Surface")
fig.colorbar(surf, ax=ax1, shrink=0.5, aspect=10)
ax1.legend()

# --- 2D Contour Plot ---
ax2 = fig.add_subplot(1, 2, 2)
contour = ax2.contourf(X, Y, Z, levels=50, cmap='viridis')
ax2.scatter(
    found_minima[:, 0], found_minima[:, 1],
    color='red', s=100, marker='*', edgecolors='white', linewidths=1.2,
    label='Global minima', zorder=5
)
for i, m in enumerate(found_minima):
    ax2.annotate(f"Min {i+1}\n({m[0]:.2f}, {m[1]:.2f})",
                 (m[0], m[1]), textcoords="offset points",
                 xytext=(10, 10), color='white', fontsize=9)
ax2.set_xlabel('x')
ax2.set_ylabel('y')
ax2.set_title("Himmelblau's Function - Contour View")
fig.colorbar(contour, ax=ax2)
ax2.legend(loc='upper right')

plt.tight_layout()
plt.show()

Number of distinct global minima found: 4
Coordinates of global minima:
  Minimum 1: x = 3.000000, y = 2.000000, f(x,y) = 0.00000000
  Minimum 2: x = -2.805118, y = 3.131313, f(x,y) = 0.00000000
  Minimum 3: x = -3.779310, y = -3.283186, f(x,y) = 0.00000000
  Minimum 4: x = 3.584428, y = -1.848127, f(x,y) = 0.00000000

Code Walkthrough

1. Defining the function and its gradient

The himmelblau() function directly implements the mathematical formula shown earlier. We also manually derived the gradient (himmelblau_grad) — the vector of partial derivatives with respect to $x$ and $y$:

$$
\frac{\partial f}{\partial x} = 4x(x^2 + y - 11) + 2(x + y^2 - 7)
$$

$$
\frac{\partial f}{\partial y} = 2(x^2 + y - 11) + 4y(x + y^2 - 7)
$$

Providing the exact gradient (instead of letting SciPy estimate it numerically) makes the optimizer converge faster and more accurately, since it avoids the overhead of finite-difference approximation.

2. Multi-start optimization

Since Himmelblau’s function has four global minima, starting the optimizer from just one point would only ever find one of them. To solve this, we:

Create a 10×10 grid of starting points spanning the domain $[-6, 6] \times [-6, 6]$ — 100 starting points in total.
Run scipy.optimize.minimize with the L-BFGS-B method (a fast quasi-Newton algorithm well suited to smooth, bounded problems) from each starting point.
Keep only results where the function value is essentially zero (result.fun < 1e-6), confirming we’ve truly hit a global minimum rather than some other stationary point.
Deduplicate results: since many nearby starting points converge to the same minimum, we check the Euclidean distance between new solutions and previously found ones, discarding near-duplicates within a 1e-4 tolerance.

This grid-based multi-start approach is a simple but effective way to perform global optimization using a fundamentally local optimizer — and because L-BFGS-B is very efficient, all 100 optimization runs complete in a fraction of a second.

3. Building the surface data

We create a fine mesh grid (200 × 200 points) over the same domain and evaluate the function at every point using vectorized NumPy operations. This gives us the Z array needed to draw a smooth 3D surface and contour map.

4. Visualization

Left panel (3D surface): Shows the overall “landscape” of the function, with two tall peaks and four valley-like basins where the function dips to zero. The red dots mark the discovered global minima, sitting exactly at the bottom of each basin.
Right panel (2D contour): A bird’s-eye view of the same landscape using color gradients (dark = low value, bright = high value). The four star markers, each labeled with its coordinates, make it immediately clear where all four minima are located relative to each other.

Together, these two views make it intuitive to see why Himmelblau’s function is such a popular test case: the four minima are spread across very different regions of the search space, forcing any global optimization algorithm to genuinely explore rather than just “roll downhill” from a single guess.

Conclusion

Himmelblau’s function beautifully illustrates a key challenge in optimization: not all problems have a unique answer. By using a multi-start strategy with SciPy’s L-BFGS-B solver and carefully deduplicating results, we successfully located all four global minima efficiently. The combination of 3D surface and 2D contour plots gives a complete, intuitive picture of the function’s structure — turning an abstract equation into something you can literally see and understand at a glance.

Minimizing the Ackley Function with Python

August 4, 2026

A Practical Global Optimization Example

Optimization problems are everywhere in engineering, machine learning, and data science, but not all objective functions are easy to minimize. Some are riddled with local minima that trap naive algorithms long before they reach the true global minimum. The Ackley function is one of the most famous benchmark functions used to test how well an optimization algorithm can escape these traps. In this article, we’ll explore the Ackley function in depth, minimize it using Python, and visualize the results in both 2D and 3D.

What Is the Ackley Function?

The Ackley function is a widely used benchmark for testing global optimization algorithms because of its nearly flat outer region combined with a large number of local minima near the center. In two dimensions, it is defined as:

$$
f(x, y) = -20 \exp\left(-0.2 \sqrt{0.5(x^2 + y^2)}\right) - \exp\left(0.5(\cos(2\pi x) + \cos(2\pi y))\right) + e + 20
$$

The global minimum is located at $(x, y) = (0, 0)$, where $f(0, 0) = 0$. What makes this function tricky is the combination of an exponential term that creates a huge, nearly flat “bowl” and a cosine term that riddles the surface with countless small local minima. A simple gradient-descent-based method will almost always get stuck in one of these local minima instead of finding the true global minimum at the origin.

Why Use a Global Optimization Algorithm?

Because of the many local minima, a gradient-based local optimizer (like scipy.optimize.minimize with BFGS) is not reliable here unless it starts very close to the global minimum. Instead, we need a global optimization algorithm. In this example, we use scipy.optimize.differential_evolution, a population-based evolutionary algorithm that explores the search space broadly before converging, making it much more robust against local minima traps.

Python Implementation

Below is the complete, self-contained source code. It defines the Ackley function, runs the global optimization, prints the results, and generates both a 3D surface plot and a 2D contour plot with the discovered minimum marked on it. The differential evolution step uses workers=-1 to parallelize across all available CPU cores, which significantly speeds up the search compared to the default single-threaded execution.

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from scipy.optimize import differential_evolution

# ---------------------------------------------------------
# 1. Define the Ackley function
# ---------------------------------------------------------
def ackley(pos, a=20, b=0.2, c=2 * np.pi):
    x, y = pos
    term1 = -a * np.exp(-b * np.sqrt(0.5 * (x**2 + y**2)))
    term2 = -np.exp(0.5 * (np.cos(c * x) + np.cos(c * y)))
    return term1 + term2 + a + np.e

# ---------------------------------------------------------
# 2. Run global optimization (Differential Evolution)
#    workers=-1 enables parallel execution for speed-up
# ---------------------------------------------------------
bounds = [(-5, 5), (-5, 5)]

result = differential_evolution(
    ackley,
    bounds,
    strategy='best1bin',
    maxiter=1000,
    popsize=20,
    tol=1e-10,
    seed=42,
    workers=-1,
    updating='deferred'
)

print("Optimization successful:", result.success)
print("Number of iterations:", result.nit)
print(f"Best solution found: x = {result.x[0]:.6f}, y = {result.x[1]:.6f}")
print(f"Minimum function value: f(x, y) = {result.fun:.10f}")

# ---------------------------------------------------------
# 3. Prepare data grid for visualization
# ---------------------------------------------------------
x_vals = np.linspace(-5, 5, 200)
y_vals = np.linspace(-5, 5, 200)
X, Y = np.meshgrid(x_vals, y_vals)
Z = ackley([X, Y])

# ---------------------------------------------------------
# 4. Plot: 3D surface + 2D contour side by side
# ---------------------------------------------------------
fig = plt.figure(figsize=(16, 7))

# --- 3D Surface Plot ---
ax1 = fig.add_subplot(1, 2, 1, projection='3d')
surf = ax1.plot_surface(X, Y, Z, cmap='viridis', edgecolor='none', alpha=0.9)
ax1.scatter(result.x[0], result.x[1], result.fun,
            color='red', s=100, marker='o', label='Global Minimum')
ax1.set_title('Ackley Function - 3D Surface')
ax1.set_xlabel('x')
ax1.set_ylabel('y')
ax1.set_zlabel('f(x, y)')
ax1.view_init(elev=35, azim=-60)
fig.colorbar(surf, ax=ax1, shrink=0.5, aspect=10)
ax1.legend()

# --- 2D Contour Plot ---
ax2 = fig.add_subplot(1, 2, 2)
contour = ax2.contourf(X, Y, Z, levels=50, cmap='viridis')
ax2.scatter(result.x[0], result.x[1], color='red', s=120,
            marker='*', edgecolor='white', linewidth=1.5,
            label=f'Minimum ({result.x[0]:.3f}, {result.x[1]:.3f})')
ax2.set_title('Ackley Function - 2D Contour Map')
ax2.set_xlabel('x')
ax2.set_ylabel('y')
fig.colorbar(contour, ax=ax2)
ax2.legend()

plt.tight_layout()
plt.show()

Optimization successful: True
Number of iterations: 123
Best solution found: x = 0.000000, y = 0.000000
Minimum function value: f(x, y) = 0.0000000000

Code Walkthrough

1. Defining the Ackley Function

The ackley() function directly implements the mathematical formula introduced earlier. It takes a 2-element array pos (representing $x$ and $y$) and returns a single scalar value. The parameters a=20, b=0.2, and c=2π are the standard constants used in the canonical definition of the function, and keeping them as arguments makes the function reusable for variations of the benchmark.

2. Global Optimization with Differential Evolution

differential_evolution maintains a population of candidate solutions and iteratively “evolves” them using mutation, crossover, and selection — a strategy inspired by biological evolution. This makes it far more resistant to getting trapped in the countless small local minima created by the cosine terms compared to gradient-based methods.

Key parameters worth understanding:

bounds: Defines the search space, here $[-5, 5]$ for both $x$ and $y$, which comfortably contains the interesting region of the function.
strategy='best1bin': A standard and reliable mutation strategy that tends to converge well on smooth, bowl-shaped benchmark functions like this one.
popsize=20: Controls how many candidate solutions are evaluated per generation. Larger values improve robustness at the cost of speed.
tol=1e-10: A tight convergence tolerance to make sure the algorithm doesn’t stop prematurely before finding a highly precise minimum.
workers=-1 and updating='deferred': These two settings together enable multi-core parallel evaluation of the population, which is the key speed-up trick here. Without them, differential_evolution evaluates each candidate solution sequentially; with them, all available CPU cores are used simultaneously, cutting runtime significantly — especially valuable if you extend this example to higher dimensions or more expensive objective functions.

After the optimization finishes, the script prints whether the run converged successfully, how many generations it took, and the best $(x, y)$ pair found along with its function value, which should be extremely close to the true global minimum of $0$ at $(0, 0)$.

3. Building the Visualization Grid

To visualize the function, we create a dense $200 \times 200$ grid of $(x, y)$ points spanning the search space using np.meshgrid, then evaluate the Ackley function across the entire grid at once using NumPy’s vectorized operations. This is far faster than looping over each point individually in Python.

4. Two Complementary Plots

3D Surface Plot: This shows the overall shape of the function — a broad, nearly flat plateau near the edges that drops sharply into a narrow, spiky funnel near the center. The red marker highlights exactly where the optimizer converged, letting you visually confirm it landed at the bottom of the funnel rather than on one of the small surrounding bumps.
2D Contour Plot: This provides a top-down view using color gradients to represent function value, making it easy to see the ring-like pattern of local minima surrounding the true global minimum. The star marker again shows the optimizer’s final solution, with its exact coordinates included in the legend.

Together, these two plots make it intuitive to understand both the global structure of the Ackley function and why gradient-based local search methods struggle with it — the small ripples visible in the contour plot near the center are individual local minima that a naive optimizer could easily get stuck in.

Conclusion

The Ackley function is a great illustration of why the choice of algorithm matters as much as the implementation when tackling non-convex optimization problems. By using differential_evolution, a population-based global optimizer, we reliably converge to the true minimum at the origin — even though the function’s surface is filled with deceptive local minima. The parallelized workers=-1 setting also demonstrates a practical, simple way to speed up evolutionary optimization runs on any multi-core machine, which becomes increasingly valuable as the problem dimensionality grows.

Solving the Rastrigin Function's Global Minimization Problem with Python

August 3, 2026

Conquering Chaos

If you’ve ever worked in optimization, you know that not all problems are created equal. Some cost functions have a single, elegant global minimum that any basic gradient descent can find in seconds. Others are a minefield of local minima designed to trap naive algorithms. Today, we’re tackling one of the most famous examples of the latter: the Rastrigin function.

This benchmark function is a rite of passage for anyone studying metaheuristic optimization — genetic algorithms, particle swarm optimization, differential evolution, and simulated annealing all get tested against it. Let’s break down exactly why it’s so difficult, and then solve it properly in Python.

What Makes the Rastrigin Function So Difficult?

The Rastrigin function is defined as:

$$
f(\mathbf{x}) = An + \sum_{i=1}^{n} \left[ x_i^2 - A\cos(2\pi x_i) \right]
$$

where $A = 10$ and $\mathbf{x} = (x_1, x_2, \dots, x_n) \in [-5.12, 5.12]^n$.

At first glance, this looks simple — it’s just a quadratic bowl ($x_i^2$) with a cosine term layered on top. But that cosine term is the troublemaker. It creates a highly regular pattern of ripples across the entire search space, generating an enormous number of local minima that grow exponentially with dimension $n$. The global minimum is always at $\mathbf{x} = \mathbf{0}$, where $f(\mathbf{0}) = 0$, but a naive local optimizer starting anywhere off-center will almost certainly get stuck in one of the surrounding “dips” long before it ever finds the true bottom.

This makes the Rastrigin function the perfect testbed for demonstrating the difference between local optimization (which gets fooled) and global optimization (which doesn’t).

Our Approach

In this article, we’ll do four things:

Visualize the Rastrigin landscape in 3D to see why it’s so treacherous.
Demonstrate how a standard local optimizer (L-BFGS-B) gets trapped in local minima.
Solve the problem properly using Differential Evolution (DE), a population-based global optimizer.
Provide a vectorized, high-speed version of the DE solver for higher-dimensional cases, since naive implementations can be painfully slow.

Let’s get into the code.

Full Source Code

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from scipy.optimize import differential_evolution, minimize
import time

# ==========================================================
# 1. Define the Rastrigin function
# ==========================================================

def rastrigin_scalar(x, A=10):
    """Standard scalar version: x is a 1D array of shape (n_dim,)"""
    x = np.asarray(x)
    n = x.shape[0]
    return A * n + np.sum(x**2 - A * np.cos(2 * np.pi * x))

def rastrigin_vectorized(X, A=10):
    """Vectorized version for fast batch evaluation.
    X has shape (n_dim, n_population) as required by scipy's
    vectorized differential_evolution."""
    return A * X.shape[0] + np.sum(X**2 - A * np.cos(2 * np.pi * X), axis=0)

# ==========================================================
# 2. Visualize the Rastrigin landscape (n = 2) in 3D and contour
# ==========================================================

bound = 5.12
res = 300
x = np.linspace(-bound, bound, res)
y = np.linspace(-bound, bound, res)
X, Y = np.meshgrid(x, y)
Z = 20 + (X**2 - 10 * np.cos(2 * np.pi * X)) + (Y**2 - 10 * np.cos(2 * np.pi * Y))

fig = plt.figure(figsize=(16, 6))

ax1 = fig.add_subplot(1, 2, 1, projection='3d')
surf = ax1.plot_surface(X, Y, Z, cmap='viridis', linewidth=0, antialiased=True, alpha=0.95)
ax1.set_title("Rastrigin Function - 3D Surface (n = 2)")
ax1.set_xlabel("x1")
ax1.set_ylabel("x2")
ax1.set_zlabel("f(x1, x2)")
ax1.view_init(elev=35, azim=45)
fig.colorbar(surf, ax=ax1, shrink=0.5, aspect=10)

ax2 = fig.add_subplot(1, 2, 2)
contour = ax2.contourf(X, Y, Z, levels=50, cmap='viridis')
ax2.plot(0, 0, 'r*', markersize=18, label='Global Minimum (0,0)')
ax2.set_title("Rastrigin Function - Contour Map")
ax2.set_xlabel("x1")
ax2.set_ylabel("x2")
ax2.legend()
fig.colorbar(contour, ax=ax2)

plt.tight_layout()
plt.show()

# ==========================================================
# 3. Demonstrate the trap: local optimization from random starts
# ==========================================================

n_trials = 30
local_minima_found = []
np.random.seed(0)

for _ in range(n_trials):
    x0 = np.random.uniform(-bound, bound, size=2)
    res_local = minimize(
        rastrigin_scalar, x0,
        method='L-BFGS-B',
        bounds=[(-bound, bound)] * 2
    )
    local_minima_found.append(res_local.fun)

plt.figure(figsize=(8, 5))
plt.hist(local_minima_found, bins=15, color='steelblue', edgecolor='black')
plt.axvline(0, color='red', linestyle='--', linewidth=2, label='True Global Minimum = 0')
plt.xlabel("Objective value found by L-BFGS-B")
plt.ylabel("Frequency (out of 30 random starts)")
plt.title("Local Optimizer Gets Trapped: Distribution of Results")
plt.legend()
plt.show()

print(f"Best value found across {n_trials} local searches: {min(local_minima_found):.4f}")
print(f"Worst value found: {max(local_minima_found):.4f}")
print(f"Success rate (f < 0.01): {sum(v < 0.01 for v in local_minima_found)}/{n_trials}")

# ==========================================================
# 4. Global optimization with Differential Evolution (10-D problem)
# ==========================================================

dim = 10
bounds = [(-bound, bound)] * dim
history_scalar = []

def callback_de(xk, convergence):
    history_scalar.append(rastrigin_scalar(xk))

start = time.time()
result_scalar = differential_evolution(
    rastrigin_scalar,
    bounds,
    strategy='best1bin',
    maxiter=1000,
    popsize=20,
    tol=1e-8,
    mutation=(0.5, 1.0),
    recombination=0.7,
    seed=42,
    callback=callback_de,
    polish=True
)
elapsed_scalar = time.time() - start

print("\n--- Standard (scalar) Differential Evolution ---")
print(f"Global minimum found: f(x*) = {result_scalar.fun:.6e}")
print(f"x* = {np.round(result_scalar.x, 4)}")
print(f"Elapsed time: {elapsed_scalar:.3f} s")
print(f"Generations run: {result_scalar.nit}")

# ==========================================================
# 5. High-speed version: vectorized Differential Evolution
# ==========================================================

start = time.time()
result_fast = differential_evolution(
    rastrigin_vectorized,
    bounds,
    strategy='best1bin',
    maxiter=1000,
    popsize=20,
    tol=1e-8,
    mutation=(0.5, 1.0),
    recombination=0.7,
    seed=42,
    polish=True,
    vectorized=True,
    updating='deferred'
)
elapsed_fast = time.time() - start

print("\n--- Vectorized (high-speed) Differential Evolution ---")
print(f"Global minimum found: f(x*) = {result_fast.fun:.6e}")
print(f"x* = {np.round(result_fast.x, 4)}")
print(f"Elapsed time: {elapsed_fast:.3f} s")
print(f"Generations run: {result_fast.nit}")
print(f"\nSpeedup factor: {elapsed_scalar / elapsed_fast:.2f}x faster")

# ==========================================================
# 6. Visualize convergence
# ==========================================================

plt.figure(figsize=(10, 5))
plt.plot(history_scalar, color='crimson', linewidth=2)
plt.yscale('log')
plt.xlabel("Generation")
plt.ylabel("Best f(x) found so far (log scale)")
plt.title(f"Convergence of Differential Evolution on {dim}-D Rastrigin Function")
plt.grid(True, which='both', linestyle='--', alpha=0.6)
plt.show()

Best value found across 30 local searches: 0.9950
Worst value found: 49.7474
Success rate (f < 0.01): 0/30

--- Standard (scalar) Differential Evolution ---
Global minimum found: f(x*) = 9.949591e-01
x* = [-0.    -0.    -0.    -0.     0.995 -0.    -0.    -0.    -0.    -0.   ]
Elapsed time: 10.238 s
Generations run: 518

--- Vectorized (high-speed) Differential Evolution ---
Global minimum found: f(x*) = 0.000000e+00
x* = [ 0. -0.  0.  0.  0.  0.  0.  0. -0.  0.]
Elapsed time: 1.981 s
Generations run: 778

Speedup factor: 5.17x faster

Code Walkthrough

1. Two flavors of the objective function

Notice that we defined two versions of the Rastrigin function: rastrigin_scalar and rastrigin_vectorized. This isn’t redundant — it’s the key to the performance story of this article.

rastrigin_scalar takes a single candidate solution (a 1D array) and returns a single number. This is the natural way to write an objective function, and it’s what most scipy.optimize routines expect by default.

rastrigin_vectorized, on the other hand, accepts an entire population of candidates at once — a 2D array where each column is one candidate solution — and returns all their objective values in one shot using NumPy’s broadcasting. This eliminates the Python-level loop overhead that occurs when an optimizer evaluates hundreds of candidates one at a time.

2. Visualizing the landscape

The 3D surface plot uses plot_surface on a 300×300 grid over the 2D search space $[-5.12, 5.12]^2$. You’ll immediately notice the “egg carton” texture — countless symmetric bumps surrounding a single, slightly deeper well at the origin. The contour map on the right shows the same thing from a bird’s-eye view, making the sheer number of local minima even more apparent. That red star marks the one true global minimum among dozens of decoys.

3. Proving the trap is real

Before jumping to the “solution,” we first prove the problem exists. We run scipy.optimize.minimize with the L-BFGS-B method (a fast, gradient-based local optimizer) from 30 different random starting points. L-BFGS-B is excellent at descending smoothly to the nearest minimum — but “nearest” is the operative word. The resulting histogram typically shows results scattered across a wide range of nonzero values, with only a small fraction landing near the true minimum of 0. This is a direct, visual demonstration of why local search alone is unreliable on this function.

4. Solving it with Differential Evolution

Differential Evolution (DE) is a population-based, gradient-free metaheuristic. Instead of following a single point downhill, it maintains an entire population of candidate solutions that evolve generation by generation through mutation, crossover, and selection. Because it explores many regions of the search space simultaneously, it’s far more resistant to getting stuck in any single local minimum.

We apply it here to a 10-dimensional version of the Rastrigin function ($n=10$) — a much harder instance than the 2D visualization, with an astronomically larger number of local minima. Key parameters:

strategy='best1bin': a classic and robust DE mutation/crossover strategy.
popsize=20: population size multiplier (actual population = popsize × dim).
mutation=(0.5, 1.0): dithering range for the mutation factor, which helps avoid premature convergence.
polish=True: after DE converges, scipy runs a quick local L-BFGS-B polish on the best solution to sharpen precision.
callback=callback_de: lets us record the best objective value after every generation, which we use later for the convergence plot.

5. The high-speed version

Standard DE evaluates the objective function once per candidate per generation using a Python-level loop internally (or optionally via multiprocessing with workers=-1, which carries process-spawning overhead that isn’t worth it for a cheap function like this one).

Instead, we use scipy’s vectorized=True mode combined with updating='deferred'. This passes the entire population to rastrigin_vectorized in a single NumPy call per generation, letting NumPy’s compiled C backend handle the heavy lifting instead of Python’s interpreter loop. For cheap-to-evaluate functions like Rastrigin, this is typically several times faster than both the naive scalar approach and multiprocessing-based parallelism, since it avoids both interpreter overhead and inter-process communication costs. The speedup factor is printed directly in the output so you can see the improvement on your own machine.

6. Reading the convergence plot

The final plot shows the best objective value found at each generation, plotted on a logarithmic y-axis (since the values shrink by orders of magnitude). You should see a characteristic staircase pattern: long flat stretches where DE is exploring without improvement, punctuated by sharp drops when it discovers a better region of the search space. By the final generations, the curve should flatten out near $10^{-8}$ to $10^{-10}$ — effectively zero, confirming that the algorithm has converged to the true global minimum at the origin.

Interpreting the Results

Once you run this in your own environment, here’s what to look for in the output:

The 3D/contour plots should confirm visually just how deceptive this landscape is — dozens of local dips surrounding one true minimum.
The local-search histogram should show that L-BFGS-B rarely finds the true minimum on its own; most runs land at nonzero values corresponding to nearby local minima.
Differential Evolution’s final result (result_scalar.fun and result_fast.fun) should be extremely close to 0, with the solution vector x* close to all zeros — even in 10 dimensions.
The speedup factor printed at the end quantifies how much faster the vectorized approach is compared to the naive scalar approach on your hardware.

Key Takeaways

The Rastrigin function is a small piece of code with an outsized lesson: gradient-based local optimizers are only as good as their starting point when the landscape is riddled with local minima. Population-based global optimizers like Differential Evolution trade some computational cost for dramatically better robustness — and with proper vectorization, that computational cost can be kept surprisingly low.

If you’re building optimization pipelines for real-world problems — hyperparameter tuning, engineering design, portfolio optimization — and you suspect your loss landscape might be non-convex or multimodal, this is exactly the kind of test you should run before trusting a purely local method.

Minimizing the Rosenbrock Function (Banana Function) with Python

August 2, 2026

What Is the Rosenbrock Function?

The Rosenbrock function is one of the most famous test problems in numerical optimization. It’s often called the “banana function” because its contour lines form a curved, banana-shaped valley. While the function looks simple, the valley is narrow and curved, which makes it deceptively difficult for many optimization algorithms to converge on efficiently.

The standard two-dimensional form is:

$$
f(x, y) = (a - x)^2 + b(y - x^2)^2
$$

with the classic parameter choice $a = 1$, $b = 100$. The global minimum sits at $(x, y) = (1, 1)$, where $f(x,y) = 0$.

The gradient, which we’ll use for faster convergence, is:

$$
\frac{\partial f}{\partial x} = -2(a - x) - 4bx(y - x^2)
$$

$$
\frac{\partial f}{\partial y} = 2b(y - x^2)
$$

Because the valley curves and is very narrow, simple gradient descent tends to zig-zag slowly along the valley floor, taking a huge number of iterations. This makes the Rosenbrock function a great case study for comparing a “naive” optimization approach against a properly optimized one.

The Approach

In this article we’ll do three things:

Implement a naive gradient descent solver from scratch in pure Python and time it.
Implement a fast, vectorized solver using scipy.optimize with the analytic gradient supplied, and time it.
Visualize both optimization paths on a 3D surface plot and a 2D contour plot, so you can literally see why one approach struggles and the other doesn’t.

Full Source Code

# ==========================================================
# Rosenbrock Function Minimization: Naive vs. Fast Approach
# ==========================================================

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D  # noqa: F401 (enables 3D projection)
from scipy.optimize import minimize
import time

# ----------------------------------------------------------
# 1. Define the Rosenbrock function and its gradient
# ----------------------------------------------------------
A = 1.0
B = 100.0

def rosenbrock(v):
    x, y = v
    return (A - x)**2 + B * (y - x**2)**2

def rosenbrock_grad(v):
    x, y = v
    dfdx = -2 * (A - x) - 4 * B * x * (y - x**2)
    dfdy = 2 * B * (y - x**2)
    return np.array([dfdx, dfdy])

# ----------------------------------------------------------
# 2. Naive gradient descent (pure Python loop, fixed step)
# ----------------------------------------------------------
def naive_gradient_descent(start, lr=0.001, n_iter=20000):
    path = [np.array(start, dtype=float)]
    x = np.array(start, dtype=float)
    for _ in range(n_iter):
        grad = rosenbrock_grad(x)
        x = x - lr * grad
        path.append(x.copy())
    return np.array(path)

start_point = np.array([-1.5, 2.0])

t0 = time.time()
naive_path = naive_gradient_descent(start_point, lr=0.001, n_iter=20000)
t1 = time.time()
naive_time = t1 - t0
naive_result = naive_path[-1]

# ----------------------------------------------------------
# 3. Fast optimization using scipy (BFGS with analytic gradient)
# ----------------------------------------------------------
bfgs_path = [start_point.copy()]

def record_path(xk):
    bfgs_path.append(xk.copy())

t2 = time.time()
res = minimize(
    rosenbrock,
    start_point,
    jac=rosenbrock_grad,
    method="BFGS",
    callback=record_path,
    options={"gtol": 1e-8}
)
t3 = time.time()
bfgs_time = t3 - t2
bfgs_path = np.array(bfgs_path)

# ----------------------------------------------------------
# 4. Print a comparison summary
# ----------------------------------------------------------
print("=" * 55)
print("Naive Gradient Descent")
print(f"  Iterations   : {len(naive_path) - 1}")
print(f"  Final point  : ({naive_result[0]:.5f}, {naive_result[1]:.5f})")
print(f"  Final f(x,y) : {rosenbrock(naive_result):.8f}")
print(f"  Elapsed time : {naive_time:.4f} sec")
print("=" * 55)
print("SciPy BFGS (analytic gradient)")
print(f"  Iterations   : {len(bfgs_path) - 1}")
print(f"  Final point  : ({res.x[0]:.5f}, {res.x[1]:.5f})")
print(f"  Final f(x,y) : {rosenbrock(res.x):.8f}")
print(f"  Elapsed time : {bfgs_time:.4f} sec")
print("=" * 55)
speedup = naive_time / bfgs_time if bfgs_time > 0 else float("inf")
print(f"BFGS was about {speedup:.1f}x faster in wall-clock time,")
print(f"and used only {len(bfgs_path)-1} iterations vs {len(naive_path)-1}.")

# ----------------------------------------------------------
# 5. Build a grid for visualization (vectorized, fast)
# ----------------------------------------------------------
x_range = np.linspace(-2, 2, 400)
y_range = np.linspace(-1, 3, 400)
X, Y = np.meshgrid(x_range, y_range)
Z = (A - X)**2 + B * (Y - X**2)**2  # fully vectorized, no Python loops

# ----------------------------------------------------------
# 6. 3D surface plot of the Rosenbrock landscape
# ----------------------------------------------------------
fig = plt.figure(figsize=(20, 8))

ax1 = fig.add_subplot(1, 2, 1, projection="3d")
surf = ax1.plot_surface(
    X, Y, np.log1p(Z),  # log1p compresses the huge range for visibility
    cmap="viridis",
    linewidth=0,
    antialiased=True,
    alpha=0.9
)
ax1.set_title("Rosenbrock Function Surface (log1p scale)", fontsize=13)
ax1.set_xlabel("x")
ax1.set_ylabel("y")
ax1.set_zlabel("log(1 + f(x, y))")
ax1.view_init(elev=35, azim=-60)
fig.colorbar(surf, ax=ax1, shrink=0.6, aspect=12, label="log(1 + f)")

# ----------------------------------------------------------
# 7. 2D contour plot with both optimization paths overlaid
# ----------------------------------------------------------
ax2 = fig.add_subplot(1, 2, 2)
levels = np.logspace(-1, 3.5, 30)
contour = ax2.contour(X, Y, Z, levels=levels, cmap="viridis", norm=None)
ax2.clabel(contour, inline=True, fontsize=6, fmt="%.0f")

ax2.plot(naive_path[:, 0], naive_path[:, 1], "r-", linewidth=1.2,
          label=f"Naive GD ({len(naive_path)-1} steps)")
ax2.plot(bfgs_path[:, 0], bfgs_path[:, 1], "b-o", linewidth=1.8,
          markersize=3, label=f"BFGS ({len(bfgs_path)-1} steps)")

ax2.plot(*start_point, "ks", markersize=8, label="Start")
ax2.plot(1, 1, "g*", markersize=16, label="True Minimum (1, 1)")

ax2.set_title("Optimization Paths on Contour Map", fontsize=13)
ax2.set_xlabel("x")
ax2.set_ylabel("y")
ax2.legend(loc="upper left", fontsize=9)
ax2.set_xlim(-2, 2)
ax2.set_ylim(-1, 3)

plt.tight_layout()
plt.show()

Code Walkthrough

Function and gradient definitions (Section 1): rosenbrock computes $f(x,y)$ directly from the formula. rosenbrock_grad computes the analytic partial derivatives shown above. Supplying the exact gradient instead of letting the optimizer estimate it numerically (via finite differences) is one of the biggest speed wins available — it avoids extra function evaluations and gives more accurate search directions.

Naive gradient descent (Section 2): This is the “textbook” implementation: at every step, move a small distance (lr = 0.001) in the direction opposite the gradient. Because the Rosenbrock valley is long, narrow, and curved, a fixed small step size forces the algorithm to take tens of thousands of tiny, zig-zagging steps just to crawl along the valley floor. We deliberately cap it at 20,000 iterations here so it finishes in reasonable time, but notice in the printed output that it still hasn’t fully converged.

Fast optimization with SciPy BFGS (Section 3): scipy.optimize.minimize with method="BFGS" uses a quasi-Newton approach: it builds up an approximation of the curvature (the Hessian) as it goes, allowing it to take much smarter, adaptively-sized steps. Combined with the analytic gradient passed via jac=rosenbrock_grad, this converges in typically fewer than 40 iterations — a dramatic reduction compared to the naive loop. The callback function records every intermediate point so we can plot the path afterward.

Timing comparison (Section 4): We use time.time() before and after each optimization to measure wall-clock performance, then print a clear side-by-side summary including how many iterations each method needed and the resulting speedup factor.

Vectorized grid for plotting (Section 5): Instead of looping over every $(x, y)$ pair in Python (which would be extremely slow for a 400×400 grid = 160,000 points), we use np.meshgrid combined with vectorized NumPy arithmetic to compute the entire surface Z in one shot. This is the same “vectorization” principle that makes the BFGS run fast — pushing work into optimized C-level array operations instead of Python-level loops.

3D surface plot (Section 6): We plot log1p(Z) instead of raw Z because the Rosenbrock function’s values span an enormous range (from 0 near the minimum to tens of thousands at the edges of the plotted domain). Without the log compression, the banana-shaped valley would be invisible, flattened under one dominant peak. The log1p (log of 1+Z) keeps the shape interpretable while safely handling values near zero.

2D contour plot with paths (Section 7): Log-spaced contour levels (np.logspace) again handle the wide value range gracefully, clearly revealing the curved valley. On top of the contours we plot the naive gradient descent path in red and the BFGS path in blue — this is where the difference between the two approaches becomes visually obvious. You’ll see the red path creeping slowly along the valley in tiny steps, while the blue path jumps directly toward the true minimum, marked with a green star at $(1, 1)$.

Result

Running the code above in Google Colab will print a timing/iteration comparison table, followed by a combined figure showing the 3D landscape on the left and the contour map with both optimization paths on the right.

=======================================================
Naive Gradient Descent
  Iterations   : 20000
  Final point  : (0.99982, 0.99964)
  Final f(x,y) : 0.00000003
  Elapsed time : 0.2666 sec
=======================================================
SciPy BFGS (analytic gradient)
  Iterations   : 38
  Final point  : (1.00000, 1.00000)
  Final f(x,y) : 0.00000000
  Elapsed time : 0.0106 sec
=======================================================
BFGS was about 25.2x faster in wall-clock time,
and used only 38 iterations vs 20000.

Takeaways

This example highlights two important optimization lessons that generalize far beyond the Rosenbrock function:

Step size matters enormously. A fixed small learning rate is “safe” but painfully slow on ill-conditioned, curved landscapes like this one.
Using derivative information and adaptive step sizing (as BFGS does) can turn tens of thousands of iterations into just a few dozen, while also landing much closer to the true minimum.

The Rosenbrock function remains a staple benchmark in optimization research precisely because it exposes these weaknesses so clearly — if an algorithm handles the banana-shaped valley efficiently, it’s usually a strong sign that it will perform well on other difficult, non-convex problems too.

Optimizing Oracle Data Aggregation

August 1, 2026

The Accuracy-Latency Trade-off in Multi-Source Price Feeds

Why This Trade-off Matters

Every DeFi protocol that liquidates a position, settles a perpetual contract, or prices a swap depends on an oracle — a system that pulls price data from multiple independent sources and produces a single, trustworthy number. The catch is that no two sources agree on both speed and quality at once. A centralized exchange tick feed arrives in tens of milliseconds but carries more microstructure noise. A TWAP feed from a decentralized exchange is smoother and harder to manipulate but arrives late. An institutional NBBO feed is extremely accurate but slow to publish.

An oracle aggregator therefore faces a genuine optimization problem: wait for more sources and get a more accurate price, or aggregate early and accept more error, but reduce the window during which the reported price is stale relative to the real market. This article builds a concrete, simulate-able model of that trade-off and finds the optimal quorum size — the number of sources the aggregator should wait for — as a function of how expensive latency is considered to be.

Modeling the Problem

Assume the true mid-price follows a simple diffusion process over the short aggregation window:

$$
dP_t = \sigma_P , dW_t
$$

Each source $i$ reports an observation $t_i$ ms after the aggregation round begins, with its own arrival-time distribution and its own measurement noise:

$$
p_i = P(t_i) + \epsilon_i, \qquad \epsilon_i \sim \mathcal{N}(0,\sigma_i^2)
$$

If the aggregator finalizes its answer at time $T$, then any source that reported at $t_i < T$ is “stale” — the true price has since drifted, and that drift itself behaves like noise:

$$
\mathrm{Var}\big[P(T)-P(t_i)\mid t_i\big] = \sigma_P^2,(T-t_i)
$$

So the effective variance of source $i$’s contribution, as seen from the aggregation deadline $T$, is

$$
\tilde\sigma_i^2(T) = \sigma_i^2 + \sigma_P^2,(T-t_i)
$$

Given a quorum of $k$ sources, the minimum-variance unbiased combination (the classical BLUE estimator) weights each source by the inverse of its effective variance:

$$
w_i = \frac{1/\tilde\sigma_i^2(T)}{\sum_{j=1}^{k} 1/\tilde\sigma_j^2(T)}, \qquad
\mathrm{MSE}(k) = \left(\sum_{i=1}^{k} \frac{1}{\tilde\sigma_i^2(T)}\right)^{-1}
$$

The aggregator’s decision variable is the quorum size $k$: waiting for the $k$-th source to arrive fixes the effective deadline $T_k$ as the $k$-th order statistic of the arrival times. The overall objective balances expected squared error against the cost of latency, priced by a coefficient $\lambda$:

$$
\mathcal{L}(k,\lambda) = \mathbb{E}[\mathrm{MSE}(k)] + \lambda,\mathbb{E}[T_k], \qquad
k^*(\lambda) = \arg\min_{k} \mathcal{L}(k,\lambda)
$$

$\lambda$ represents how expensive one additional millisecond of latency is, expressed in the same variance units as the pricing error — a higher $\lambda$ corresponds to applications like liquidation engines or MEV-sensitive settlement, where staleness is costly; a lower $\lambda$ suits applications like slow-moving collateral valuation.

Simulation Design

Six representative sources are modeled, each with a Gamma-distributed arrival time (shape 2) and a fixed measurement noise level:

Source	Mean latency	Noise std
CEX Fast-Tick A	40 ms	$0.50
CEX Fast-Tick B	60 ms	$0.80
DEX TWAP (5-block)	90 ms	$1.20
DEX TWAP (30-block)	130 ms	$1.80
Institutional NBBO	180 ms	$0.30
Aggregator Node	260 ms	$2.50

For each of 20,000 Monte Carlo trials, arrival times are drawn independently per source, sorted, and the closed-form BLUE variance formula above is applied directly to the first $k$ arrivals — no explicit noise sampling is needed, because the expected MSE has an exact analytical expression given the arrival times. This keeps the whole simulation vectorized and avoids the far more expensive approach of drawing noise realizations and averaging squared errors numerically.

Source Code

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D  # noqa: F401

plt.style.use('dark_background')
rng = np.random.default_rng(42)

# ---- Source configuration ----
source_names = np.array([
    "CEX Fast-Tick A",
    "CEX Fast-Tick B",
    "DEX TWAP (5-block)",
    "DEX TWAP (30-block)",
    "Institutional NBBO",
    "Aggregator Node"
])
n_sources = len(source_names)

mean_latency_ms = np.array([40.0, 60.0, 90.0, 130.0, 180.0, 260.0])
gamma_shape = 2.0
latency_scale = mean_latency_ms / gamma_shape

sigma_i = np.array([0.50, 0.80, 1.20, 1.80, 0.30, 2.50])   # USD, per-source noise std
sigma2_i = sigma_i ** 2

sigma_price = 0.02   # USD per sqrt(ms), diffusion coefficient of the true mid-price
M = 20000            # Monte Carlo trials

# ---- Simulate arrival times per source ----
arrival_times = np.empty((M, n_sources))
for i in range(n_sources):
    arrival_times[:, i] = rng.gamma(shape=gamma_shape, scale=latency_scale[i], size=M)

argsort_idx = np.argsort(arrival_times, axis=1)
sorted_t = np.take_along_axis(arrival_times, argsort_idx, axis=1)
sorted_sigma2 = sigma2_i[argsort_idx]

# ---- Closed-form BLUE aggregation for each quorum size k ----
mean_mse = np.empty(n_sources)
mean_T = np.empty(n_sources)

for k in range(1, n_sources + 1):
    T_k = sorted_t[:, k - 1]
    t_first_k = sorted_t[:, :k]
    sigma2_first_k = sorted_sigma2[:, :k]
    staleness_var = sigma_price ** 2 * (T_k[:, None] - t_first_k)
    total_var = sigma2_first_k + staleness_var
    inv_var_sum = np.sum(1.0 / total_var, axis=1)
    mse_trials = 1.0 / inv_var_sum
    mean_mse[k - 1] = mse_trials.mean()
    mean_T[k - 1] = T_k.mean()

k_values = np.arange(1, n_sources + 1)

# ---- Latency cost sweep ----
lambda_grid = np.logspace(-4, -1, 40)   # USD^2 per ms
loss_grid = mean_mse[:, None] + lambda_grid[None, :] * mean_T[:, None]
k_star_idx = np.argmin(loss_grid, axis=0)
k_star = k_values[k_star_idx]

# ---- Console summary ----
print(f"{'Source':<22}{'Mean latency (ms)':>20}{'Noise std ($)':>16}")
for name, lat, s in zip(source_names, mean_latency_ms, sigma_i):
    print(f"{name:<22}{lat:>20.1f}{s:>16.2f}")
print()
print(f"{'k':>3}{'E[latency] (ms)':>20}{'E[MSE] ($^2)':>16}")
for k, t, m in zip(k_values, mean_T, mean_mse):
    print(f"{k:>3}{t:>20.2f}{m:>16.4f}")

# ---- Figure 1: 2D accuracy-latency Pareto frontier ----
fig1 = plt.figure(figsize=(9, 6))
ax1 = fig1.add_subplot(111)
ax1.plot(mean_T, mean_mse, 'o-', color='#00e5ff', linewidth=2, markersize=8)
for k, t, m in zip(k_values, mean_T, mean_mse):
    ax1.annotate(f'k={k}', (t, m), textcoords="offset points", xytext=(8, 6),
                 color='white', fontsize=10)
ax1.set_xlabel('Expected latency  E[T_k]  (ms)', fontsize=11)
ax1.set_ylabel('Expected squared error  E[MSE]  (USD²)', fontsize=11)
ax1.set_title('Accuracy vs. Latency Pareto Frontier by Quorum Size', fontsize=13)
ax1.grid(alpha=0.3)
plt.tight_layout()
plt.show()

# ---- Figure 2: 3D loss surface over quorum size and latency cost ----
K_grid, L_grid = np.meshgrid(k_values, lambda_grid, indexing='ij')
logL_grid = np.log10(L_grid)

fig2 = plt.figure(figsize=(11, 8))
ax2 = fig2.add_subplot(111, projection='3d')
ax2.xaxis.pane.set_facecolor((0.08, 0.08, 0.1, 1.0))
ax2.yaxis.pane.set_facecolor((0.08, 0.08, 0.1, 1.0))
ax2.zaxis.pane.set_facecolor((0.08, 0.08, 0.1, 1.0))
surf = ax2.plot_surface(K_grid, logL_grid, loss_grid, cmap='plasma',
                         edgecolor='none', alpha=0.92)
ax2.scatter(k_star, np.log10(lambda_grid),
            loss_grid[k_star_idx, np.arange(len(lambda_grid))],
            color='white', s=35, depthshade=True, label='optimal k*(λ)')
ax2.set_xlabel('Quorum size k', fontsize=10, labelpad=10)
ax2.set_ylabel('log10(λ)  [latency cost weight]', fontsize=10, labelpad=10)
ax2.set_zlabel('Expected total loss', fontsize=10, labelpad=10)
ax2.set_title('Expected Total Loss Surface over Quorum Size and Latency Cost',
              fontsize=13, pad=20)
fig2.colorbar(surf, shrink=0.55, aspect=12, pad=0.1, label='Expected total loss')
ax2.legend(loc='upper left')
ax2.view_init(elev=25, azim=-60)
plt.tight_layout()
plt.show()

# ---- Figure 3: Optimal quorum size regime map ----
fig3 = plt.figure(figsize=(9, 5))
ax3 = fig3.add_subplot(111)
ax3.step(np.log10(lambda_grid), k_star, where='post', color='#ff6ec7', linewidth=2)
ax3.set_xlabel('log10(λ)  [latency cost weight, USD²/ms]', fontsize=11)
ax3.set_ylabel('Optimal quorum size  k*', fontsize=11)
ax3.set_yticks(k_values)
ax3.set_title('Optimal Quorum Size as a Function of Latency Cost', fontsize=13)
ax3.grid(alpha=0.3)
plt.tight_layout()
plt.show()

Code Walkthrough

Source generation. Each source’s arrival time is drawn from a Gamma(shape=2, scale=mean/2) distribution, which produces a realistic right-skewed latency profile — most reports arrive close to the mean, but occasional stragglers arrive much later, just like real network and block-inclusion delays.

Sorting once, reusing everywhere. np.argsort on the arrival-time matrix gives, for every trial, the order in which sources reported. np.take_along_axis and fancy indexing then reorder both the arrival times and the corresponding noise variances into that same order. This means the first $k$ columns of sorted_t and sorted_sigma2 always represent “the $k$ fastest sources to report in this trial” — exactly what an aggregator with quorum $k$ would have used.

Why no noise sampling is needed. Rather than drawing $\epsilon_i$ explicitly and averaging squared errors over many draws (which would need either a second random dimension or far more trials to converge), the code uses the closed-form BLUE variance formula $\mathrm{MSE}(k) = \left(\sum 1/\tilde\sigma_i^2\right)^{-1}$ directly. Since this expression only depends on the arrival times, a single Monte Carlo dimension over arrival times is sufficient. This is the “fast” version of the simulation — it is exact in expectation and avoids an extra order of magnitude of computation.

Linear separability of the loss. Because $\mathbb{E}[\mathrm{MSE}(k) + \lambda T_k] = \mathbb{E}[\mathrm{MSE}(k)] + \lambda,\mathbb{E}[T_k]$, the code only needs to compute mean_mse and mean_T once per quorum size — the entire sweep over 40 values of $\lambda$ is then a single vectorized outer-product-style computation (loss_grid), with no additional simulation required.

Reading the Results

Source                   Mean latency (ms)   Noise std ($)
CEX Fast-Tick A                       40.0            0.50
CEX Fast-Tick B                       60.0            0.80
DEX TWAP (5-block)                    90.0            1.20
DEX TWAP (30-block)                  130.0            1.80
Institutional NBBO                   180.0            0.30
Aggregator Node                      260.0            2.50

  k     E[latency] (ms)    E[MSE] ($^2)
  1               24.36          0.8568
  2               45.84          0.2654
  3               73.47          0.1667
  4              113.06          0.1261
  5              178.40          0.0951
  6              323.92          0.0808

Figure 1 — the Pareto frontier. Each point on this curve is a quorum size $k=1,\dots,6$. Moving right along the curve trades higher expected latency for lower expected squared error. The steepness between consecutive points shows where quorum increases are “cheap” (large error reduction for modest latency) versus where they become “expensive” (small error reduction, but a large latency jump, typically once the slow institutional and aggregator-node sources must be included).

Figure 2 — the 3D loss surface. This is the full picture of $\mathcal{L}(k,\lambda)$. Along the $k$-axis, the surface curves down initially (adding sources reduces variance) then curves back up (added sources contribute mostly latency once diminishing accuracy returns kick in). Along the $\lambda$-axis, the entire surface tilts — as latency becomes more expensive (larger $\lambda$, right side of the plot), the valley of the surface shifts toward smaller $k$. The white markers trace the ridge of optimal quorum choices $k^*(\lambda)$ across that tilt, visually confirming that there is no single “best” quorum size — it depends entirely on how costly staleness is for the specific application consuming the price feed.

Figure 3 — the regime map. This collapses the ridge from Figure 2 into a direct decision rule: for a given latency cost $\lambda$, what quorum size should the aggregator configure? The step structure shows discrete regimes — at very low $\lambda$ (latency nearly free), the optimum sits near $k=6$, using every source for maximum accuracy. As $\lambda$ increases, the optimal quorum steps down, eventually settling near $k=1$–$2$ once latency dominates the objective, favoring only the fastest sources even at a real accuracy cost.

Practical Implications

This framework maps directly onto real oracle architecture decisions. Protocols serving latency-sensitive functions — perpetual futures mark prices, liquidation triggers — sit on the high-$\lambda$ end of the regime map and should favor small, fast quorums, accepting more noise per update but compensating with higher update frequency. Protocols serving slower functions — collateral factor recalculation, governance-parameter feeds — sit on the low-$\lambda$ end and should wait for larger quorums including slower, high-quality sources like TWAPs and institutional feeds.

The same structure also explains why many production oracle systems (Chainlink-style networks included) use tiered aggregation: a fast layer with a small quorum for time-critical consumers, and a slower, larger-quorum layer for consumers where accuracy dominates. The model above gives a quantitative way to choose the quorum size for each tier rather than setting it by intuition alone, and the closed-form BLUE approach means the entire optimization can be recomputed cheaply whenever source latency or noise characteristics change — for instance, after adding a new exchange feed or observing degraded reliability from an existing one.

Cross-Chain Bridge Liquidity & Fee Optimization

July 31, 2026

Minimizing Cost When Moving Assets Across Chains

Bridging assets between blockchains is rarely a single, clean transaction. Every bridge route is backed by a liquidity pool with finite depth, and every unit of value you push through that pool causes price impact (slippage) on top of the base fee. If you naively send your entire transfer through a single “cheapest-looking” bridge, you can end up paying far more than necessary — especially for large transfers. The smarter approach is to split the transfer across multiple routes so that the marginal cost of each route stays balanced. This is directly analogous to the classic water-filling problem in information theory and portfolio allocation.

In this article we build a concrete, numerical example: moving a stablecoin-pegged asset from Ethereum to BSC through three independent bridge routes (via Arbitrum, Optimism, and Polygon aggregator liquidity), each modeled as a constant-product AMM pool with its own reserves and fee. We’ll derive the optimal allocation analytically, implement a fast closed-form solver, benchmark it against brute-force grid search, and visualize the results in both 2D and 3D.

1. The Model

Each bridge route $i$ is modeled as a constant-product AMM pool with input reserve $R_{in,i}$, output reserve $R_{out,i}$, and fee rate $f_i$. If we send amount $a$ through route $i$, the effective input after fees is:

$$a_{\text{eff}} = a(1-f_i)$$

and the output received is:

$$\Delta y_i(a) = \frac{R_{out,i} \cdot a_{\text{eff}}}{R_{in,i} + a_{\text{eff}}}$$

The cost of routing amount $a$ through route $i$ (in input-equivalent units) is:

$$C_i(a) = a - \Delta y_i(a)$$

Given a total transfer amount $A$ that we want to split across $n$ routes as $(a_1, a_2, \dots, a_n)$, the optimization problem is:

$$\min_{a_1,\dots,a_n} \sum_{i=1}^{n} C_i(a_i) \quad \text{s.t.} \quad \sum_{i=1}^{n} a_i = A, \quad a_i \ge 0$$

Since $\Delta y_i(a)$ is concave and increasing (diminishing returns from slippage), this is a convex optimization problem. The Karush-Kuhn-Tucker (KKT) condition tells us that at the optimum, the marginal price (marginal output per marginal input) must be equal across all active routes:

$$\frac{d(\Delta y_i)}{da}\Big|_{a_i^*} = \lambda \quad \text{for all active } i$$

where $\lambda$ is a shared shadow price. This is the classic water-filling condition. The marginal price function for a constant-product pool is:

$$p_i(a) = \frac{d(\Delta y_i)}{da} = \frac{R_{out,i}(1-f_i)R_{in,i}}{\left(R_{in,i} + a(1-f_i)\right)^2}$$

This is monotonically decreasing in $a$, which lets us invert it in closed form to get $a_i(\lambda)$, and then find the $\lambda$ that satisfies $\sum_i a_i(\lambda) = A$ using bisection — no iterative numerical optimizer needed. This closed-form + bisection approach is dramatically faster than a generic nonlinear solver or grid search, which matters if you want to re-optimize routing on every block.

2. Full Source Code

Run this single cell in Google Colaboratory.

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D  # noqa: F401 (enables 3D projection)
import time

# ----------------------------------------------------------------------
# 1. Route definitions
#    Three independent bridge routes from Ethereum -> BSC, each exposed
#    as an aggregate constant-product liquidity pool (Rin, Rout, fee).
# ----------------------------------------------------------------------
routes = [
    {"name": "Route A (via Arbitrum)", "Rin": 500_000, "Rout": 498_000, "fee": 0.0005},
    {"name": "Route B (via Optimism)", "Rin": 300_000, "Rout": 299_000, "fee": 0.0008},
    {"name": "Route C (via Polygon)",  "Rin": 800_000, "Rout": 795_000, "fee": 0.0003},
]
route_params = [(r["Rin"], r["Rout"], r["fee"]) for r in routes]
route_names  = [r["name"] for r in routes]
n_routes = len(routes)

# ----------------------------------------------------------------------
# 2. Core AMM math
# ----------------------------------------------------------------------
def amm_output(a, Rin, Rout, fee):
    """Output amount received when sending `a` through a constant-product pool."""
    a = np.asarray(a, dtype=float)
    a_eff = a * (1 - fee)
    return Rout * a_eff / (Rin + a_eff)

def marginal_price(a, Rin, Rout, fee):
    """Marginal output per marginal input (derivative of amm_output)."""
    a_eff = a * (1 - fee)
    return Rout * (1 - fee) * Rin / (Rin + a_eff) ** 2

def inverse_marginal(lmbda, Rin, Rout, fee):
    """Given a target marginal price lambda, solve for the input amount a
    such that marginal_price(a) = lambda. Returns 0 if lambda exceeds the
    pool's maximum marginal price (i.e. the route should not be used)."""
    p0 = Rout * (1 - fee) / Rin  # marginal price at a = 0
    if lmbda >= p0:
        return 0.0
    a_eff = np.sqrt(Rout * (1 - fee) * Rin / lmbda) - Rin
    return max(a_eff / (1 - fee), 0.0)

# ----------------------------------------------------------------------
# 3. Water-filling solver (fast closed-form + bisection)
# ----------------------------------------------------------------------
def total_alloc(lmbda, params):
    return sum(inverse_marginal(lmbda, *p) for p in params)

def water_filling(A, params, iters=100):
    hi = max(Rout * (1 - fee) / Rin for (Rin, Rout, fee) in params)
    lo = 1e-9
    for _ in range(iters):
        mid = 0.5 * (lo + hi)
        if total_alloc(mid, params) > A:
            lo = mid
        else:
            hi = mid
    lmbda = 0.5 * (lo + hi)
    allocs = np.array([inverse_marginal(lmbda, *p) for p in params])
    s = allocs.sum()
    if s > 0:
        allocs = allocs * (A / s)  # normalize to exactly match A
    return allocs, lmbda

def cost_of_allocation(allocs, params):
    total_out = sum(amm_output(a, *p) for a, p in zip(allocs, params))
    return allocs.sum() - total_out

# ----------------------------------------------------------------------
# 4. Baseline strategies for comparison
# ----------------------------------------------------------------------
def naive_best_single(A, params):
    costs = [A - amm_output(A, *p) for p in params]
    idx = int(np.argmin(costs))
    return costs[idx], idx

def equal_split_cost(A, params):
    a_each = A / len(params)
    total_out = sum(amm_output(a_each, *p) for p in params)
    return A - total_out

# ----------------------------------------------------------------------
# 5. Brute-force grid search (slow reference method for benchmarking)
# ----------------------------------------------------------------------
def brute_force_grid(A, params, n_grid=400):
    a1 = np.linspace(0, A, n_grid)
    a2 = np.linspace(0, A, n_grid)
    A1, A2 = np.meshgrid(a1, a2)
    A3 = A - A1 - A2
    valid = A3 >= 0
    out1 = amm_output(A1, *params[0])
    out2 = amm_output(A2, *params[1])
    out3 = amm_output(np.clip(A3, 0, None), *params[2])
    cost = A - (out1 + out2 + out3)
    cost = np.where(valid, cost, np.inf)
    idx = np.unravel_index(np.argmin(cost), cost.shape)
    return cost[idx], (A1[idx], A2[idx], A3[idx])

# ----------------------------------------------------------------------
# 6. Speed benchmark: water-filling vs brute-force grid search
# ----------------------------------------------------------------------
A_bench = 1_000_000

t0 = time.perf_counter()
opt_allocs, lmbda_star = water_filling(A_bench, route_params)
opt_cost = cost_of_allocation(opt_allocs, route_params)
t1 = time.perf_counter()

grid_cost, grid_allocs = brute_force_grid(A_bench, route_params, n_grid=400)
t2 = time.perf_counter()

print("=== Speed & Accuracy Benchmark (A = {:,}) ===".format(A_bench))
print(f"Water-filling (closed-form):  {(t1 - t0)*1000:.4f} ms | cost = {opt_cost:,.4f}")
print(f"Brute-force grid (400x400):   {(t2 - t1)*1000:.4f} ms | cost = {grid_cost:,.4f}")
print(f"Speedup: {(t2 - t1) / (t1 - t0):,.1f}x faster, with lower (more accurate) cost\n")

for name, a in zip(route_names, opt_allocs):
    print(f"  {name:30s} -> allocate {a:,.2f} ({a/A_bench*100:5.2f}%)")

# ----------------------------------------------------------------------
# 7. Sweep total amount A and compare strategies
# ----------------------------------------------------------------------
A_values = np.linspace(10_000, 2_000_000, 200)
opt_bps, naive_bps, equal_bps, savings_pct = [], [], [], []

for A in A_values:
    allocs, _ = water_filling(A, route_params)
    c_opt = cost_of_allocation(allocs, route_params)
    c_naive, _ = naive_best_single(A, route_params)
    c_equal = equal_split_cost(A, route_params)

    opt_bps.append(c_opt / A * 1e4)
    naive_bps.append(c_naive / A * 1e4)
    equal_bps.append(c_equal / A * 1e4)
    savings_pct.append((c_naive - c_opt) / c_naive * 100)

# ----------------------------------------------------------------------
# 8. Diagram: bridge route topology (Ethereum -> BSC via 3 routes)
# ----------------------------------------------------------------------
fig0, ax0 = plt.subplots(figsize=(10, 6))
ax0.set_xlim(-1, 11)
ax0.set_ylim(-3.5, 3.5)
ax0.axis("off")

ax0.scatter([0], [0], s=2200, color="#3b82f6", zorder=3)
ax0.text(0, 0, "Ethereum", ha="center", va="center", color="white", fontsize=11, fontweight="bold", zorder=4)
ax0.scatter([10], [0], s=2200, color="#f59e0b", zorder=3)
ax0.text(10, 0, "BSC", ha="center", va="center", color="white", fontsize=12, fontweight="bold", zorder=4)

y_positions = [2.2, 0, -2.2]
colors = ["#10b981", "#8b5cf6", "#ef4444"]
for r, y, c in zip(routes, y_positions, colors):
    ax0.annotate(
        "", xy=(9.3, y), xytext=(0.7, 0),
        arrowprops=dict(arrowstyle="-|>", color=c, lw=2.5,
                         connectionstyle=f"arc3,rad={y*0.05}")
    )
    label = f'{r["name"]}\nRin={r["Rin"]:,}  Rout={r["Rout"]:,}  fee={r["fee"]*100:.2f}%'
    ax0.text(5, y + (0.55 if y >= 0 else -0.75), label, ha="center", fontsize=9, color=c)

ax0.set_title("Bridge Route Topology: Ethereum → BSC (3 Parallel Liquidity Routes)", fontsize=13)
plt.tight_layout()
plt.show()

# ----------------------------------------------------------------------
# 9. 2D chart: cost (bps) and savings (%) vs transfer amount
# ----------------------------------------------------------------------
fig1, (axL, axR) = plt.subplots(1, 2, figsize=(14, 5.5))

axL.plot(A_values, naive_bps, label="Best single route (naive)", color="#ef4444", lw=2)
axL.plot(A_values, equal_bps, label="Equal split (naive)", color="#f59e0b", lw=2, linestyle="--")
axL.plot(A_values, opt_bps, label="Water-filling (optimal split)", color="#10b981", lw=2.5)
axL.set_xlabel("Transfer amount A")
axL.set_ylabel("Cost (basis points)")
axL.set_title("Routing Cost vs Transfer Size")
axL.legend()
axL.grid(alpha=0.3)

axR.plot(A_values, savings_pct, color="#3b82f6", lw=2.5)
axR.fill_between(A_values, 0, savings_pct, color="#3b82f6", alpha=0.15)
axR.set_xlabel("Transfer amount A")
axR.set_ylabel("Cost savings vs naive best single route (%)")
axR.set_title("Savings from Optimal Multi-Route Splitting")
axR.grid(alpha=0.3)

plt.tight_layout()
plt.show()

# ▼ここに実行結果画像(2Dチャート)を挿入▼
# (leave space in the blog layout here for the pasted screenshot)

# ----------------------------------------------------------------------
# 10. 3D chart: cost surface over (a1, a2) split, for fixed A
# ----------------------------------------------------------------------
A_fixed = 1_000_000
n_grid = 150
a1_range = np.linspace(0, A_fixed, n_grid)
a2_range = np.linspace(0, A_fixed, n_grid)
A1, A2 = np.meshgrid(a1_range, a2_range)
A3 = A_fixed - A1 - A2

out1 = amm_output(A1, *route_params[0])
out2 = amm_output(A2, *route_params[1])
out3 = amm_output(np.clip(A3, 0, None), *route_params[2])
cost_surface = A_fixed - (out1 + out2 + out3)
cost_surface = np.where(A3 >= 0, cost_surface, np.nan)

opt_allocs_fixed, _ = water_filling(A_fixed, route_params)
opt_cost_fixed = cost_of_allocation(opt_allocs_fixed, route_params)

fig2 = plt.figure(figsize=(11, 8))
ax2 = fig2.add_subplot(111, projection="3d")
surf = ax2.plot_surface(A1, A2, cost_surface, cmap="viridis", alpha=0.85,
                         linewidth=0, antialiased=True)
ax2.scatter([opt_allocs_fixed[0]], [opt_allocs_fixed[1]], [opt_cost_fixed],
            color="red", s=90, label="Water-filling optimum", depthshade=False)

ax2.set_xlabel("a1: amount via Route A")
ax2.set_ylabel("a2: amount via Route B")
ax2.set_zlabel("Total cost")
ax2.set_title(f"Total Routing Cost Surface (A = {A_fixed:,}, a3 = A - a1 - a2)")
fig2.colorbar(surf, shrink=0.6, aspect=12, label="Total cost")
ax2.legend()
plt.tight_layout()
plt.show()

3. Code Walkthrough

Section 1 — Route definitions. We model three bridge routes from Ethereum to BSC, each as a tuple of (Rin, Rout, fee) representing the aggregate liquidity depth exposed by that route (a bridge aggregator abstraction). Route A and B have shallower liquidity and thus more slippage per unit of size; Route C has the deepest liquidity but the lowest raw marginal price at zero.

Section 2 — Core AMM math. amm_output implements the standard constant-product formula $\Delta y = \frac{R_{out} \cdot a(1-f)}{R_{in} + a(1-f)}$. marginal_price is its analytic derivative — the instantaneous exchange rate you get for an infinitesimally small additional unit sent through the pool. inverse_marginal algebraically inverts this derivative, letting us ask “what input amount gives this route a marginal price of exactly $\lambda$?” in closed form, with no iterative root-finding required for a single route.

Section 3 — Water-filling solver. water_filling performs bisection search over the shared shadow price $\lambda$. For each candidate $\lambda$, total_alloc sums up how much each route would receive under that price using the closed-form inverse. Because total_alloc is monotonically decreasing in $\lambda$, standard bisection converges quickly (100 iterations is far more than enough for double-precision accuracy). Once $\lambda^*$ is found, allocations are rescaled slightly to make sure they sum to exactly $A$ despite floating-point rounding.

Section 4 — Baselines. naive_best_single evaluates sending the entire amount through each individual route and picks the cheapest one — this mirrors how many simple bridge UIs behave today. equal_split_cost is a naive “just split evenly across routes” heuristic, useful to show that even simple splitting isn’t as good as marginal-price-aware splitting.

Section 5 — Brute-force grid search. This is the “slow” reference implementation, included purely for benchmarking. It evaluates the cost function on a dense 400×400 grid over $(a_1, a_2)$ (with $a_3$ implied), fully vectorized with NumPy. Even vectorized, it is orders of magnitude slower and less precise than the closed-form water-filling approach, because its accuracy is capped by grid resolution.

Section 6 — Speed benchmark. We directly time both methods on the same $A = 1{,}000{,}000$ transfer and print the wall-clock time and resulting cost for each. This demonstrates concretely why the closed-form approach is the one to use for time-sensitive scenarios like on-chain routing or real-time UI quotes: it’s both faster and more accurate.

Section 7 — Sweep over transfer size. We compute the cost (in basis points) of all three strategies across a range of transfer amounts from 10,000 up to 2,000,000, along with the percentage savings of water-filling over the naive best-single-route strategy.

Sections 8–10 — Visualizations. These build the topology diagram, the 2D cost/savings charts, and the 3D cost surface, described in detail below.

4. Visualizing the Results

4.1 Bridge Route Topology

The first chart is a simple schematic of the three parallel liquidity routes between Ethereum and BSC, annotated with each route’s reserves and fee. It’s a useful mental model before diving into the numbers: think of these as three separate pipes of different diameters (liquidity depth) between the two chains.

4.2 Cost vs Transfer Size, and Savings from Splitting

The left panel plots routing cost in basis points against transfer size for all three strategies. Notice how the “best single route” (red) and “equal split” (orange, dashed) curves rise steeply as the transfer size grows — because pushing a large amount through one pool (or splitting it blindly in half/thirds) causes disproportionate slippage. The water-filling curve (green) stays consistently lower because it dynamically shifts more volume toward whichever route currently has the best marginal price, and stops adding to a route the moment its marginal price drops below what another route still offers.

The right panel converts this into a direct savings percentage: for small transfers, all three routes have similar marginal prices at low volume, so the benefit of splitting is small. As the transfer size grows, the gap widens substantially, since a single pool alone would suffer heavy slippage that splitting avoids entirely.

4.3 The 3D Cost Surface

The 3D surface shows total cost as a function of how much is sent through Route A ($a_1$) and Route B ($a_2$), with Route C absorbing the remainder ($a_3 = A - a_1 - a_2$) for a fixed total transfer of 1,000,000 units. The bowl-like shape confirms convexity: cost rises sharply near the edges of the triangular domain (where one route is forced to absorb almost everything, causing heavy slippage) and dips toward a minimum somewhere in the interior. The red marker shows the exact point found by the water-filling algorithm — sitting precisely at the bottom of the bowl, confirming that the closed-form solution matches the true numerical minimum.

4.4 Benchmark Output

The printed benchmark output (from Section 6 of the code) shows the wall-clock time for the closed-form water-filling method versus the brute-force grid search, along with the resulting allocation across the three routes for a 1,000,000-unit transfer.

=== Speed & Accuracy Benchmark (A = 1,000,000) ===
Water-filling (closed-form):  0.7574 ms | cost = 387,863.9679
Brute-force grid (400x400):   9.3112 ms | cost = 387,864.1008
Speedup: 12.3x faster, with lower (more accurate) cost

  Route A (via Arbitrum)         -> allocate 312,904.33 (31.29%)
  Route B (via Optimism)         -> allocate 187,889.00 (18.79%)
  Route C (via Polygon)          -> allocate 499,206.67 (49.92%)

5. Takeaways

The core insight is that cross-chain routing cost isn’t just about comparing headline fees — it’s a convex optimization problem driven by AMM slippage curves. The optimal strategy equalizes marginal price across all active routes rather than chasing the lowest average price of a single route, and for constant-product pools this equalization condition has a closed-form solution that can be solved via simple bisection instead of a general-purpose optimizer. This makes it practical to re-solve the allocation in real time — for example inside a bridge aggregator’s quoting engine — every time liquidity conditions or the requested transfer size change.

A Spring Chain Model in Python

1. The Physical Model

2. Full Python Source Code

3. Code Walkthrough

Physical model functions

Naive loop vs. vectorized grid computation

Three independent solving methods

Visualization

4. Interpreting the Results

5. Extending the Model

A Global Optimization Walkthrough

What Is the Goldstein-Price Function?

Strategy: Global Search First, Local Refinement Second

The Source Code

Code Walkthrough

Visualizing the Function

Graph Explanation

A Journey Through a Deceptively Simple Optimization Landscape

Introduction

The Mathematics of the Beale Function

Optimization Strategy

Source Code

Code Walkthrough

Visualizing the Optimization Landscape

Interpreting the Visualization

Conclusion

A Clean Introduction to Convex Optimization

Introduction

Mathematical Definition

Python Implementation

Code Explanation

Visualization and Interpretation

Conclusion

A Journey Through Multiple Global Minima

Introduction

What is Himmelblau’s Function?

Strategy

Full Source Code

Code Walkthrough

1. Defining the function and its gradient

2. Multi-start optimization

3. Building the surface data

4. Visualization

Conclusion

A Practical Global Optimization Example

What Is the Ackley Function?

Why Use a Global Optimization Algorithm?

Python Implementation

Code Walkthrough

1. Defining the Ackley Function

2. Global Optimization with Differential Evolution

3. Building the Visualization Grid

4. Two Complementary Plots

Conclusion

Conquering Chaos

What Makes the Rastrigin Function So Difficult?

Our Approach

Full Source Code

Code Walkthrough

1. Two flavors of the objective function

2. Visualizing the landscape

3. Proving the trap is real

4. Solving it with Differential Evolution

5. The high-speed version

6. Reading the convergence plot

Interpreting the Results

Key Takeaways

What Is the Rosenbrock Function?

The Approach

Full Source Code

Code Walkthrough

Result

Takeaways

The Accuracy-Latency Trade-off in Multi-Source Price Feeds

Why This Trade-off Matters

Modeling the Problem

Simulation Design

Source Code

Code Walkthrough

Reading the Results