Comparing Anomaly Alerts With Human-Coded Incident Records
Compare anomaly alerts with human-coded incident records only after fixing the incident definition, observation window and matching rule. Measure agreement between human reviewers separately from errors against their final reference labels. Otherwise, a disagreement about what counts as an incident can be mistaken for a detector failure.