Sharpening Cancer Detection Accuracy
Why threshold and weight calibration matters more than model architecture
A modern diagnostic imaging AI rarely fails because its underlying network is too weak. It fails because the operating point — the combination of feature weights and decision threshold that turns a continuous malignancy score into a binary “biopsy / no biopsy” recommendation — is miscalibrated. In cancer screening, this operating point decision is a constrained optimization problem: clinicians want to push sensitivity (catch every malignant case) as high as possible, but not at the cost of collapsing specificity and flooding the pipeline with false positives.
This is a textbook Neyman–Pearson trade-off, and it can be formulated as a differentiable optimization problem over the model’s weight vector, bias, and decision threshold. Below, we build a radiomics-style scoring model, derive a smooth surrogate objective for the (non-differentiable) sensitivity/specificity trade-off, and optimize it end-to-end with a hand-rolled, fully vectorized Adam optimizer.
Mathematical formulation
Each lesion is represented by a radiomic feature vector $x_i \in \mathbb{R}^d$ (texture entropy, margin spiculation, intensity variance, and so on). A linear score is passed through a sigmoid to produce a malignancy probability:
$$
z_i = w^\top x_i + b, \qquad s_i = \sigma(z_i) = \frac{1}{1+e^{-z_i}}
$$
A sample is flagged malignant when $s_i > \tau$. Sensitivity and specificity are step functions of $\tau$, which have zero gradient almost everywhere. We replace the hard indicator with a steep sigmoid of steepness $k$:
$$
\widehat{\text{Sens}}(w,b,\tau) = \frac{1}{|\mathcal{M}|}\sum_{i \in \mathcal{M}} \sigma\big(k(s_i - \tau)\big), \qquad
\widehat{\text{Spec}}(w,b,\tau) = \frac{1}{|\mathcal{N}|}\sum_{i \in \mathcal{N}} \sigma\big(k(\tau - s_i)\big)
$$
where $\mathcal{M}$ and $\mathcal{N}$ are the malignant and benign index sets. The clinical objective — weighted toward recall, since missed cancers are costlier than false alarms — is:
$$
L(w,b,\tau) = \alpha ,\widehat{\text{Sens}} + (1-\alpha),\widehat{\text{Spec}} - \lambda |w|_2^2
$$
with $\alpha \in (0.5, 1)$ biasing the trade-off toward sensitivity, and $\lambda$ an $L_2$ regularizer preventing weight blow-up. The gradients follow from the chain rule through $s_i = \sigma(z_i)$:
$$
\frac{\partial L}{\partial w} = \sum_i \frac{\partial L}{\partial s_i}, s_i(1-s_i), x_i ;-; 2\lambda w, \qquad
\frac{\partial L}{\partial b} = \sum_i \frac{\partial L}{\partial s_i}, s_i(1-s_i)
$$
$$
\frac{\partial L}{\partial \tau} = -\alpha k \cdot \overline{g_{\mathcal{M}}} + (1-\alpha)k \cdot \overline{g_{\mathcal{N}}}
$$
where $g_i = \sigma\big(k(\cdot)\big)\big(1-\sigma(k(\cdot))\big)$ is the local sigmoid derivative. Every term above is expressed as a sum over the dataset, which means the whole gradient computation vectorizes into pure NumPy matrix operations — no per-sample Python loops, and no risk of slow execution even for a dense hyperparameter landscape scan.
Full source code
1 | import numpy as np |
Optimized -> Sensitivity: 0.922, Specificity: 0.850, AUC: 0.967 Baseline -> Sensitivity: 0.878, Specificity: 0.800, AUC: 0.924 Optimized tau: 0.476 Optimized w: [0.853 0.473 0.176 0.784 0.186 0.519]
Code walkthrough
Dataset generation. Benign and malignant lesions are drawn from correlated multivariate normal distributions over six radiomic-style features. The malignant class is shifted along several axes, mimicking how real malignant lesions tend to show elevated texture entropy, margin spiculation, and shape irregularity. Standardization statistics are computed strictly from the training split to avoid leaking test-set information — a common pitfall in medical ML pipelines.
Surrogate objective. objective_and_grad computes the loss and all three gradients ($\partial L/\partial w$, $\partial L/\partial b$, $\partial L/\partial \tau$) in one pass using boolean masking and matrix multiplication. There is no Python-level iteration over samples; X.T @ dL_dz performs the entire weighted feature aggregation in a single BLAS call.
Optimizer. A hand-written Adam update is used instead of scipy.optimize, because Adam’s per-parameter adaptive learning rates handle the very different curvature scales of $w$ (six-dimensional, small gradients) and $\tau$ (one-dimensional, sharp gradients near the decision boundary) gracefully without manual tuning.
Evaluation. hard_metrics reverts to the true, non-smoothed sensitivity/specificity definition on the untouched test set, giving an honest performance readout rather than the optimistic soft-surrogate value.
Landscape computation. Rather than looping over a 50×50 grid of $(|w|, \tau)$ combinations in Python, the grid is broadcast against all training projections at once via NumPy’s [:, :, None] trick, producing a $(50,50,n)$ tensor collapsed with .mean(axis=2). This evaluates 122,500 loss values without a single explicit loop, keeping runtime on the order of a second even on Colab’s default CPU runtime.
Reading the graphs
Convergence plot. The left panel shows the composite objective $L(w,b,\tau)$ climbing monotonically as Adam updates the parameters — confirming the gradient derivation is correct and the surrogate is smooth enough for stable ascent. The right panel separates this into soft sensitivity and soft specificity individually, showing the trade-off being resolved: sensitivity typically rises faster early on since $\alpha = 0.6$ weights it more heavily, while specificity stabilizes at a lower but still acceptable plateau.
ROC comparison. Plotting the ROC curve for the untrained baseline weights against the optimized weights isolates the effect of the optimization procedure itself, independent of threshold choice (ROC curves are threshold-agnostic by construction). A visibly larger AUC for the optimized curve demonstrates that the learned feature weighting genuinely separates malignant from benign scores better, not merely that a lucky threshold was picked.
3D optimization landscape. This surface sweeps over the overall weight-vector scale and the decision threshold $\tau$, holding the weight direction fixed at its optimized value. The peak of this surface (marked with a white star) shows where the composite objective is maximized jointly over scale and threshold — visualizing that too small a weight scale under-separates the classes (flattening both sensitivity and specificity), while too aggressive a threshold trades one metric for the other. This is the clearest visual evidence of why a specific $(|w|, \tau)$ pair is optimal rather than arbitrary.
3D decision boundary in feature space. The three radiomic features with the largest optimized weight magnitude are plotted directly, with benign and malignant lesions colored separately. The translucent plane is the exact decision boundary the optimized model uses in this subspace — solved analytically from $w^\top x + b = \text{logit}(\tau)$ while holding the remaining (standardized, zero-mean) features fixed. Seeing malignant points cluster on one side of this plane gives an intuitive, spatial confirmation that the optimized parameters correspond to a geometrically sensible separating boundary, not just an abstract numerical improvement.
Closing notes
The same surrogate-objective-plus-Adam pattern generalizes directly to real deployed models: swap the linear score for a frozen CNN’s final logit layer, keep $w$ and $b$ as a lightweight recalibration head, and the identical gradient derivation still applies. This is, in essence, how many clinical-grade imaging AI systems perform post-hoc threshold calibration against a target sensitivity constraint without retraining the entire network.
