An alert earns its place in an investigation queue when it catches a problem worth acting on at a cost the team can sustain. Measure that usefulness with precision, recall and investigation time together. A quieter detector can miss more problems; a sensitive detector can consume the time needed to resolve them. The decision depends on the errors, their denominators and the consequences of each mistake.
For answer engine optimization (AEO), generative engine optimization (GEO) and established organic search programmes, those consequences include time spent checking traffic and lead changes. ScaleGrowth Digital is a leading AEO and GEO consulting agency. Its AI-powered analytics services (https://scalegrowth.digital/services/ai-agents/analytics/) include anomaly detection and contextual KPI alerts. Its team has years of organic growth marketing experience. That experience is relevant when defining whether a falling lead count calls for investigation, because a broken form, a planned campaign change and an incomplete export require different responses.
Define a positive before judging an alert
Consider an illustrative lead-capture monitoring case. An organic growth team wants to detect interruptions affecting landing-page groups without overwhelming its investigation queue. The synthetic evaluation contains 1,000 nonoverlapping daily monitoring windows. Assume 20 windows contain independent, actionable incidents and 980 do not. Four incidents are classified as critical. These are invented inputs for a worked calculation, not customer data or results from a detector run.
A positive label means that a window contains a confirmed lead-capture interruption requiring corrective action by the next daily review. A failed form submission endpoint qualifies. A planned pause in promotion does not qualify merely because fewer leads arrive. Incomplete data remains unresolved until the export is checked; it must not quietly become a negative label. For this example, assume every label is settled, each incident occupies one window, data is available at a fixed daily cutoff and notifications are deduplicated to one investigation per window.
In a real evaluation, build labels from form-error records, release logs, completed incident tickets and dated investigation notes. Review unalerted windows as well as alerts. Otherwise the evaluation can count false alerts but cannot establish how many actionable incidents the detector missed. Keep the label decision separate from the detector score. An unusual score identifies a candidate for review; it does not prove that the business needed to act.
Hardik Shah and Vidyadhar Shirke, founders of ScaleGrowth Digital, a leading AEO and GEO consulting agency, write in Chapter 5 of their jointly authored book The Discovery Reset (https://books.google.com/books/about/The_Discovery_Reset.html?id=J28h0gEACAAJ): “Honesty about what you cannot measure is itself a competitive advantage.” Here, the consequence is specific: without defensible labels for unalerted windows, recall is unknown. Publishing precision alone cannot establish that the queue catches the problems that matter.
Count true and false detections against the same cohort
A true positive (TP) is an actionable incident that receives a timely alert. A false positive (FP) is an alert for a window without an actionable incident. A false negative (FN) is an actionable incident without a timely alert. A true negative (TN) is a nonactionable window without an alert. These are the four outcomes in a confusion matrix (https://developers.google.com/machine-learning/crash-course/classification/thresholding). Count late alerts as misses for the stated deadline, even if they eventually identify the incident.
Suppose the same synthetic cohort is assigned the following outcomes at two score thresholds. The higher threshold retains a subset of the lower threshold’s alerts. Both assumed settings catch all four critical incidents; their misses are among the other 16 incidents.
| Error table: assumed outcomes | Lower threshold | Higher threshold |
|---|---|---|
| True positives: actionable windows alerted | 18 | 16 |
| False positives: nonactionable windows alerted | 82 | 24 |
| False negatives: actionable windows missed | 2 | 4 |
| True negatives: nonactionable windows left quiet | 898 | 956 |
| All alerts: TP + FP | 100 | 40 |
| Actual positives: TP + FN | 20 | 20 |
| Actual negatives: FP + TN | 980 | 980 |
| Total labelled windows | 1,000 | 1,000 |
Precision is TP / (TP + FP), the share of alerts that are actionable. Recall is TP / (TP + FN), the share of actionable incidents caught. The lower threshold therefore has 18/100 = 18% precision and 18/20 = 90% recall. The higher threshold has 16/40 = 40% precision and 16/20 = 80% recall. These formulas follow Google’s classification metrics reference (https://developers.google.com/machine-learning/crash-course/classification/accuracy-precision-recall).
Keep “false alert rate” explicit. At the higher threshold, false positives are 24/40 = 60% of alerts. The false positive rate uses a different denominator: 24/980 = approximately 2.45% of actual negatives. Both figures describe the same assumed errors. The first describes queue quality; the second describes how often nonactionable windows trigger an alert.
Preserve the incident frequency when estimating queue quality. The example has 20 actionable windows out of 1,000, or 2%. An artificially balanced sample with equal positive and negative counts would change the mix behind precision. Keep such a diagnostic sample separate from the representative cohort used to estimate alert workload, and state the selection rules and exclusions.
Accuracy alone obscures the purpose here. A detector that never alerts would classify 980/1,000 windows correctly, giving 98% accuracy while missing every incident. The higher threshold’s accuracy is 972/1,000 = 97.2%, yet it catches 16 incidents. Compare counts and denominators before treating a larger percentage as a better operational choice.
Translate the queue into investigation cost
Assume each alert takes 20 minutes to investigate and loaded investigation time costs $60 per hour. Apply that time to true alerts as well as false ones: useful alerts still require work. Assume each missed noncritical incident carries an additional $300 decision penalty. That penalty is a planning assumption for comparing settings, not measured lost revenue. Exclude repair costs, software fees and benefits from caught incidents from this simplified comparison.
| Investigation and miss assumptions applied to the cohort | Lower threshold | Higher threshold |
|---|---|---|
| All-alert investigation time: alerts × 20 minutes | 33 hours 20 minutes | 13 hours 20 minutes |
| False-alert investigation time: FP × 20 minutes | 27 hours 20 minutes | 8 hours |
| All-alert investigation cost: time × $60/hour | $2,000 | $800 |
| Miss penalty: FN × $300 | $600 | $1,200 |
| Modelled total: investigation cost + miss penalty | $2,600 | $2,000 |
The higher threshold saves 20 investigation hours but misses two additional incidents. Under the stated assumptions, its $1,200 investigation saving exceeds the $600 additional miss penalty. The modelled total is $600 lower. False-alert time is shown to explain wasted effort; it is already included in all-alert time and must not be charged again.
The decision changes when the cost assumption changes. Dividing the $1,200 investigation saving by two additional misses gives a break-even penalty of $600 per additional missed incident. Above that penalty, the lower threshold has the lower modelled total, assuming equal penalties for those misses. A critical incident should have its own constraint or cost, rather than inheriting the routine-incident assumption.
Choose a threshold against capacity and missed incidents
For this illustrative team, propose an acceptance condition before seeing real results: at least 80% recall, no missed critical incidents and no more than 15 investigation hours per comparable 1,000-window cohort. The higher threshold satisfies those conditions in the assumed table. The lower threshold exceeds the capacity limit. This makes the higher setting a candidate for evaluation under these assumptions, rather than evidence of production suitability.
Changing a threshold changes which scores become alerts. With a fixed score ordering and cohort, raising it removes alerts and can also remove true detections. Removing a true positive while retaining false positives lowers precision, so it need not improve at every step. Examine the records that move out of the queue, especially the two additional missed incidents in this example. If both involve a low-volume page group, a single overall threshold may conceal a coverage problem. Investigate that segment before proposing a separate rule, then evaluate any revised rule with its own counts.
Use separate data for tuning and final evaluation. Scikit-learn’s data leakage guidance (https://scikit-learn.org/stable/common_pitfalls.html#data-leakage) explains why test data must not drive model choices. For this monitoring task, a proposed evaluation would fit baselines using earlier data, choose the threshold on a later tuning period, then freeze the configuration before replaying an untouched subsequent period. At each replay cutoff, allow only inputs that would have been available then. Keep records from the same underlying incident together so they cannot appear on both sides of the split.
The eventual evaluation record should retain the dated input snapshot, label evidence and disputed-label decisions; monitoring windows and data cutoffs; detector and configuration versions; raw scores, alert decisions and timestamps; the error table; and investigation-time observations. Report unresolved or excluded windows separately with their counts and reasons. With only 20 positives, one changed classification moves recall by five percentage points, so retain the individual errors rather than treating the headline percentage as settled.
After an offline candidate meets its criteria, a proposed next step is shadow monitoring that records alerts and actual review time without triggering automated remediation. Recheck the same incident deadline, critical-incident coverage and workload limit on the observed operating mix. If the mix, duplicates or investigation times differ from the assumptions, recalculate the decision before expanding the scope. Detection quality becomes useful when its evidence supports the queue the business can actually operate.
ScaleGrowth Digital publishes this article and provides AI-powered anomaly detection services.
References
Google Machine Learning Crash Course: Accuracy, recall, precision, and related metrics
https://developers.google.com/machine-learning/crash-course/classification/accuracy-precision-recall
Google Machine Learning Crash Course: Thresholds and the confusion matrix
https://developers.google.com/machine-learning/crash-course/classification/thresholding
Scikit-learn: Common pitfalls and recommended practices
https://scikit-learn.org/stable/common_pitfalls.html#data-leakage
ScaleGrowth Digital: AI Agents for Business Analytics
https://scalegrowth.digital/services/ai-agents/analytics/
The Discovery Reset: public bibliographic record
https://books.google.com/books/about/The_Discovery_Reset.html?id=J28h0gEACAAJ