Balancing Storage Cost, Incident Investigation, and Compliance
How long should you keep your logs? Keep them too briefly and you can’t investigate an incident or pass an audit. Keep them too long and storage costs balloon. This post turns the question into an optimization problem and solves it with Python.
The Problem
A company collects six log sources. Each has a daily volume $v_s$ (GB/day), a compliance minimum retention $T_{\min,s}$, and a weight $w_s$, the probability that the source is needed in an incident investigation.
| Source | GB/day | Compliance minimum (days) | Weight $w_s$ |
|---|---|---|---|
| Firewall | 400 | 90 | 0.5 |
| DNS | 250 | 90 | 0.4 |
| Web/App | 300 | 90 | 0.6 |
| Auth/IdP | 40 | 365 | 0.9 |
| EDR | 150 | 180 | 0.8 |
| Cloud Audit | 60 | 365 | 0.8 |
Each source has two decision variables:
- $h$: days kept in hot storage (fast, searchable, expensive)
- $T$: total retention in days, with the remaining $T-h$ days in cold archive (cheap, slow to restore)
The Model
Storage cost. With hot price $p_h$ and cold price $p_c$ in dollars per GB-month, the steady-state annual cost is
$$C_{\text{stor}}(h,T)=12,v,\bigl[p_h,h+p_c,(T-h)\bigr]$$
Dwell time. The time between a compromise and its detection, $D$, is modeled as log-normal:
$$F(d)=P(D\le d)=\Phi!\left(\frac{\ln d-\mu}{\sigma}\right),\qquad \mu=\ln(\text{median})$$
Investigation risk. With $\lambda$ relevant incidents per year, a restore penalty $r$ when the needed log is only in cold storage, and a loss $L$ when the log has already been deleted:
$$R(h,T)=\lambda,w,\Bigl[r,\bigl(F(T)-F(h)\bigr)+L,\bigl(1-F(T)\bigr)\Bigr]$$
Optimization problem.
$$\min_{h,T}; C(h,T)=C_{\text{stor}}(h,T)+R(h,T)\quad\text{s.t.}\quad T_{\min}\le T\le T_{\max},;; h_{\min}\le h\le T$$
Here $T_{\max}$ is a data-minimization cap and $h_{\min}$ is the shortest hot window needed for day-to-day monitoring.
Optimality conditions. Setting the partial derivatives to zero, with $f=F’$ the dwell-time density, gives
$$f(T^*)=\frac{12,v,p_c}{\lambda,w,(L-r)},\qquad f(h^*)=\frac{12,v,(p_h-p_c)}{\lambda,w,r}$$
The first condition says to keep logs in cold storage until the marginal storage cost equals the marginal reduction in expected loss. The second says the same for the hot window, using the restore penalty.
Speed-up. The cost separates into $C(h,T)=a(h)+b(T)$ with
$$a(h)=12,v,(p_h-p_c),h-\lambda,w,r,F(h),\qquad b(T)=12,v,p_c,T+\lambda,w,\bigl[(r-L)F(T)+L\bigr]$$
For each $T$, the best $h$ is the prefix minimum $\min_{h\le T}a(h)$, which one cumulative-minimum pass computes. This removes the two-dimensional search.
The Source Code
1 | import time |
Code Walkthrough
1. Parameters
SOURCES holds the six log sources with daily volume, compliance minimum, and investigation weight. The constants define the economics and the threat model:
P_HOTandP_COLDare storage prices in dollars per GB-month.LAMBDAis the number of relevant incidents per year.RESTOREis the extra cost when the needed log must be rehydrated from cold storage (delay and engineering time).LOSSis the cost of a “blind” incident, where the logs were already deleted.MEDIAN_DWELLandSIGMAdefine the log-normal dwell-time distribution. The heavy right tail is what makes long retention valuable.H_MIN,H_MAX, andT_MAXare the search bounds.T_MAXacts as a data-minimization cap, so the optimizer can never recommend keeping data forever.
2. Model
dwell_cdf is the log-normal CDF, computed with scipy.special.ndtr, the standard normal CDF. The np.where guard returns 0 for non-positive days, so the function accepts scalars, vectors, and 2D grids without special cases.
cost_surface implements the two cost components from the formulas above. It relies on NumPy broadcasting: pass H as a column and T as a row, and it returns the whole $(h,T)$ cost grid at once with no loops.
optimize_bruteforce evaluates every feasible $(h,T)$ pair and masks infeasible ones (h < H_MIN or h > T) with infinity. It is simple and obviously correct, so it serves as the reference.
optimize_fast uses the separable structure $C=a(h)+b(T)$:
np.minimum.accumulate(a)gives the best hot window for every possible $T$ in one pass.- The
argline recovers which $h$ achieved that minimum. Becausea == pmis true wherever a new running minimum appears, taking a cumulative maximum over those indices tracks the latest minimizer. - The final
argminoverpm[k] + bis a one-dimensional search.
The assert in the solving section checks that both methods find the same minimum cost for every source, so the fast version is verified rather than assumed.
evaluate returns the storage and risk components for any chosen policy, which makes policy comparisons easy.
3. Solving and Comparing
The script times both solvers and prints the speed-up. It then builds the optimal-policy table and compares five strategies:
- A: the compliance minimum, all hot.
- B: one year for everything, all hot.
- C: the compliance minimum with a 30-day hot window.
- D: one year with a 30-day hot window.
- E: the optimized tiered policy.
The last block re-solves the whole problem for an 8×8 grid of median dwell times and blind-incident losses. This is only practical because the fast solver is so cheap.
4. Visualization
All six panels are drawn into one figure and shown with a single plt.show(). They are explained in the results section below.
Execution Results
Brute force: 29.9 ms | Fast: 3.6 ms | Speed-up: 8.3x
GB/day T_min hot_days total_days storage_$/yr risk_$/yr total_$/yr
source
Firewall 400.0 90.0 26 535 22,252.8 32,133.2 54,386.0
DNS 250.0 90.0 36 586 17,400.0 21,734.6 39,134.6
Web/App 300.0 90.0 46 634 25,027.2 27,937.7 52,964.9
Auth/IdP 40.0 365.0 218 730 11,447.0 11,522.7 22,969.8
EDR 150.0 180.0 99 730 22,363.2 20,565.4 42,928.6
Cloud Audit 60.0 365.0 168 730 13,714.6 12,922.6 26,637.2
storage_$/yr 112,204.8
risk_$/yr 126,816.3
total_$/yr 239,021.1
dtype: float64
storage_$/yr risk_$/yr total_$/yr
policy
A: Compliance min (all hot) 178,800.0 354,460.0 533,260.0
B: 365 d (all hot) 525,600.0 97,315.1 622,915.1
C: Compliance min (30 d hot) 48,624.0 509,527.0 558,151.0
D: 365 d (30 d hot) 62,496.0 286,668.1 349,164.1
E: Optimized tiered 112,204.8 126,816.3 239,021.1

Interpreting the Results
Optimal policy per source. The optimizer recommends very different policies for different sources:
- Firewall, DNS, and Web/App are high-volume with modest weights. Their hot windows are short (about 26, 36, and 46 days), but their total retention is far beyond the 90-day minimum, at roughly 535, 586, and 634 days. The cold tier costs so little that extending retention is cheap insurance against the long tail of slow detections.
- Auth/IdP, EDR, and Cloud Audit are low-volume, high-weight sources. They hit the 730-day cap, and their hot windows are much longer (about 218, 99, and 168 days) because a restore delay on these logs is expensive and keeping them hot is cheap in absolute terms.
Policy comparison. The optimized tiered policy costs about $239k per year in total. The alternatives are much worse:
| Policy | Total ($/yr) |
|---|---|
| A: compliance minimum, all hot | about 533k |
| B: 365 days, all hot | about 623k |
| C: compliance minimum, 30 days hot | about 558k |
| D: 365 days, 30 days hot | about 349k |
| E: optimized tiered | about 239k |
Policies A and C look cheap on storage but carry very large expected risk. Policy B cuts risk but pays heavily for hot storage. Even policy D, which is a reasonable-sounding rule of thumb, costs roughly 46% more than the optimum. The lesson is that neither “keep less” nor “keep everything hot” wins. The savings come from matching each source’s tiering and retention to its volume, value, and the shape of your detection-time distribution.
3D cost surface (top left) and contour view (top right). For the firewall, the surface is a valley with a clear minimum, marked in red.
- Along the $T$ axis, cost falls steeply at first as the risk of blind incidents drops, then rises slowly as cold storage accumulates.
- Along the $h$ axis, cost rises gently with hot storage but also falls as restore penalties are avoided, which produces the shallow bowl around the optimum.
- Cutting off the left side of the surface is the minimum hot window, and cutting off the diagonal is the rule $h\le T$.
- The contour view shows the same minimum from above. The valley is much flatter along $T$ than along $h$, so a retention period near the optimum is forgiving while the hot window deserves more careful tuning.
Cost trade-off curve (bottom left). Storage cost grows linearly with $T$, while expected risk decays as the log-normal tail is covered. Their sum is the black curve, whose minimum sits at $T^*$. This is a direct picture of the condition $f(T^*)=12,v,p_c/\bigl(\lambda,w,(L-r)\bigr)$: the optimum is where the density of “late discoveries” no longer justifies another day of archive.
Sensitivity surface (top middle). The optimal total cost rises with both the median dwell time and the loss per blind incident.
- Slower detection pushes more incidents into the tail, so more retention and more cost are needed.
- Larger blind-incident losses make long retention more valuable, so the optimal spend grows.
- Reducing your median detection time lowers the cost of the optimal policy, which is a business case for investing in detection engineering.
Per-source bar chart (bottom middle). The compliance minimum is a floor, not a target. For every source, the optimal total retention is at or above it, and for most it is far above. The chart also shows how much the hot window varies, from under a month to over seven months.
Stacked policy chart (bottom right). Each bar splits storage from expected risk. Policies at the extremes are dominated by one component, and the optimized policy balances the two.
Conclusion
Log retention is not a single number to apply across the board. By modeling detection delay statistically and pricing storage tiers explicitly, the problem becomes a small, well-behaved optimization with a clean closed-form intuition. The separable structure also lets you solve it fast enough to run sensitivity analyses over the assumptions that matter most.











