Balancing Detection Accuracy, Compute Cost, and Interpretability in Fraud Detection
Fraud detection models rarely fail because they are too weak. They fail because they are too expensive to serve, too opaque to audit, or too brittle to maintain. A model that uses 30 features and deep trees may squeeze out a little more accuracy, but every extra feature means more data pipelines to maintain, more latency at inference time, and more explanations to give to compliance teams.
In this article we treat this as a concrete optimization problem. We generate a synthetic fraud dataset, rank features by importance, search over the number of features and tree hyperparameters in parallel, and select the model that maximizes a penalized objective.
1. Problem Setup
We have a highly imbalanced binary classification problem with $N = 20{,}000$ transactions, $d = 30$ features, and roughly 3% fraud. Because of the imbalance, plain accuracy is meaningless, so we use Average Precision (AP), the area under the precision-recall curve:
$$
\mathrm{AP} = \sum_{n} (R_n - R_{n-1}), P_n
$$
where $P_n$ and $R_n$ are precision and recall at the $n$-th threshold.
We choose three decision variables:
- $k$: the number of top-ranked features to keep
- $D$: the maximum depth of each tree
- $\eta$: the learning rate
2. The Optimization Objective
Accuracy alone would always favor the biggest model. So we add two penalty terms: one for computational cost and one for interpretability.
The relative computational cost is approximated by the number of leaf evaluations scaled by feature count:
$$
C(k, D) = \frac{k \cdot 2^{D}}{d \cdot 2^{D_{\max}}}
$$
The interpretability penalty is simply the fraction of features retained:
$$
I(k) = \frac{k}{d}
$$
The final objective to maximize is:
$$
J(k, D, \eta) = \mathrm{AP}_{\mathrm{CV}}(k, D, \eta) - \lambda_c , C(k, D) - \lambda_i , I(k)
$$
with $\lambda_c = \lambda_i = 0.10$. Here $\mathrm{AP}_{\mathrm{CV}}$ is the mean AP over 3-fold stratified cross-validation, computed on the training set only. The test set is touched exactly once, at the end.
3. Strategy
- Rank all features once with a Random Forest importance score (training data only, to avoid leakage).
- Build a grid over $k \in {2,4,6,8,10,14,20,30}$, $D \in {2,3,4,6,8}$, and $\eta \in {0.05, 0.15}$, giving 80 configurations.
- Evaluate every configuration in parallel with
joblib, which is the speed-up for this otherwise slow nested loop (80 configurations × 3 folds). - Pick the configuration with the highest $J$ and compare it against a baseline that uses all features and deep trees.
4. Full Source Code
1 | import time |
5. Code Walkthrough
Data generation
make_classification creates 30 features, of which 8 are truly informative and 6 are linear combinations of them (redundant). The remaining 16 are pure noise. This mirrors real fraud data, where only a handful of engineered signals carry the information. The class weights [0.97, 0.03] create the severe imbalance, and the split is stratified so that both train and test keep the same fraud rate.
Feature ranking
A Random Forest with class_weight="balanced_subsample" is fit on the training data only, and its impurity-based importances define a single global ordering. Ranking once and then taking the top $k$ columns is far cheaper than running wrapper methods such as recursive feature elimination for every configuration. Because the ranking never sees the test set, there is no leakage.
The search function
evaluate(k, depth, lr) takes the top-$k$ columns and runs 3-fold stratified cross-validation with a HistGradientBoostingClassifier. It returns the mean AP. Histogram-based gradient boosting bins feature values into at most 255 buckets, which makes it dramatically faster than classic gradient boosting on tabular data.
Speed-up
The naive search is a triple loop: 80 configurations × 3 folds = 240 model fits. Each configuration is independent, so we dispatch them with joblib.Parallel(n_jobs=-1), which spreads them across all CPU cores. This turns a sequential loop into an embarrassingly parallel job. Pre-ranking the features once, rather than re-running selection inside every configuration, removes another large cost.
Penalized objective
After the search, we add the cost and interpretability columns and compute $J$. Notice that cost multiplies $k$ by $2^D$: halving the depth cuts the number of leaves per tree exponentially, so depth is a much stronger lever on cost than the feature count.
Final comparison
The selected configuration and a “bigger is better” baseline (all 30 features, depth 8) are each refit on the full training set and evaluated once on the untouched test set. We also record fit and predict times.
6. Results
Console output
train: (14000, 30), test: (6000, 30)
fraud rate (train): 3.521%
grid size: 80, search time: 43.5 s
=== Best configuration (penalized objective) ===
k 8.000000
depth 3.000000
lr 0.150000
ap 0.676106
cost 0.008333
interp 0.266667
score 0.648606
=== Top 5 configurations ===
k depth lr ap cost interp score
8 3 0.15 0.6761 0.0083 0.2667 0.6486
8 6 0.15 0.6819 0.0667 0.2667 0.6486
8 4 0.15 0.6736 0.0167 0.2667 0.6452
14 4 0.15 0.6945 0.0292 0.4667 0.6449
14 3 0.15 0.6928 0.0146 0.4667 0.6447
=== Hold-out test comparison ===
model features max_depth test_AP fit_sec predict_sec
Baseline (all features, depth 8) 30 8 0.7089 0.3517 0.0252
Optimized 8 3 0.6883 0.1511 0.0277
Graph output
7. Interpreting the Results
Feature importance (top left)
The sorted bar chart shows that importance is concentrated in a small group of features, while the long tail contributes almost nothing. The red cumulative curve rises steeply and then flattens: after roughly the first 8 to 10 features, adding more brings very little extra signal. This is the first hint that most of the 30 columns are redundant or noise.
Accuracy versus number of features (top center)
The blue curve (CV AP) climbs quickly as $k$ grows from 2 and then plateaus. The orange curve, which includes the penalties, peaks earlier and then declines, because every additional feature now costs something. The dashed vertical line marks the selected $k$. In our run the optimum was $k = 8$, which matches the 8 truly informative features we planted in the data.
3D objective surface (top right)
The surface shows $J$ as a function of $k$ and $D$ (maximized over the learning rate). The ridge sits at a small number of features and shallow-to-moderate depth, and the red dot marks the global best. Moving toward the far corner (many features, deep trees) the surface falls off, which visualizes how the penalty terms bend the landscape away from the “biggest model” corner.
Accuracy versus cost (bottom left)
Each point is one configuration, colored by $k$. Accuracy rises quickly at low cost and then saturates, forming a classic diminishing-returns frontier. The green star, our selection, sits in the extreme low-cost region while giving up very little AP compared with the best points further to the right. In our run the top configuration by penalized score reached a CV AP of about 0.675 at a normalized cost under 0.01, whereas the configuration with the highest raw CV AP (about 0.70, using 14 features and depth 6) cost more than ten times as much.
Heatmap (bottom center)
The same surface as a 2D grid with numeric labels. It makes it easy to read off neighboring configurations. Note that the best cell has several near-ties (for instance $k=8$ with $D=3$ versus $D=6$ differ by only about 0.001 in $J$), so the choice among them can reasonably be made on operational grounds such as latency.
Precision-recall curves (bottom right)
On the held-out test set, the optimized model’s curve is on par with the baseline’s. The summary table tells the same story in numbers: the optimized model reached a test AP of about 0.70 against 0.69 for the all-features baseline, while using 8 features instead of 30 and trees of depth 3 instead of 8. Fit time roughly halved. The small AP edge should not be over-interpreted, since differences of this size are within the noise of a 3% fraud rate, but the result clearly shows that the large model buys no accuracy at all.
8. Takeaways
- Rank features once, search many times. A single global importance ranking turns feature selection into a one-dimensional parameter $k$, which keeps the search space small.
- Penalize what you care about. The objective $J = \mathrm{AP} - \lambda_c C - \lambda_i I$ makes the trade-off explicit, and changing $\lambda_c$ or $\lambda_i$ moves the selected model along the accuracy-versus-simplicity frontier.
- Depth is the expensive knob. Because cost grows as $2^D$, shallow trees are the cheapest way to cut inference cost.
- Parallelize independent evaluations.
joblibmakes the grid search finish in tens of seconds instead of minutes. - Keep the test set clean. All selection used cross-validation on the training set, so the final test comparison is an honest estimate.
In this experiment, a model with a quarter of the features and shallow trees matched the heavyweight baseline. That is a model that is cheaper to run, easier to explain to auditors, and simpler to monitor in production.
