Chapter 9 · Reading 6 of 10
Building alerts analysts can trust
Security+ SY0-801 · Objectives 4.4
The night shift receives eighty alerts and escalates two. The day shift receives eight alerts and escalates two. Does the second rule work better? Counts tell you how much work the analysts saw. They leave out the incidents that never raised an alert.
Trace an alert backward to collection
Define the behavior of interest, the data source and the expected response. Syslog can carry device events. SNMP polling can supply counters and device state. Flow records summarize conversations. A mirrored port can feed a packet analyzer, subject to capacity and encryption limits. An endpoint agent adds process context. Agentless collection can reduce installation overhead while depending on credentials, network reachability and polling intervals.
Aggregation collects events. Parsing exposes fields that a rule can use. Normalization aligns different representations of identity, time and severity. Retain original records so a parser error can be found. Track last-seen time per source, time from event to ingestion, dropped events and parsing failures. A dashboard with zero alerts may describe a quiet shift or a broken collector. An independent heartbeat distinguishes those possibilities.
Tune with known positives and negatives
A rule raises alerts when a privileged role appears on a new device. Maintenance creates predictable noise. Narrow the exception to approved time, device and owner, then replay labeled examples. Precision is true alerts divided by all alerts raised. Recall is detected true cases divided by all known true cases. A rule can improve precision by becoming so narrow that important incidents disappear. Review both measurements and the consequences of each error.
For the sample in the figure, distinguish a missed incident from a false alarm. Examine the missing event chains and seek an independent source before changing the threshold. Archive the rule version and test set. Give each alert a runbook, an owner and enough context for the next shift to reproduce the triage decision.
Count effort alongside accuracy
Four false alarms in a low-volume queue may take five minutes to clear. Four false alarms that repeatedly wake the only on-call identity engineer impose a different operational cost. Record what evidence the analyst needs to distinguish a real incident and whether that information is already available at alert time. Adding a stable asset owner or current maintenance ticket can improve triage even before the detection logic changes.
Also ask what the missed incidents have in common. If all occurred on a platform with no reliable agent, adjusting the threshold on existing events may have little effect. The right intervention could be improving collection. Keep detection quality, collection quality and response capacity visible as separate decisions.
Compare the cost of a decision
Take a labeled sample of important events along with ordinary maintenance traffic. Run the candidate rule against both. If suppressing expected overnight jobs also hides a genuine incident, document which signal would distinguish those cases. A narrow exception based on a specific job identity and scheduled window may preserve useful coverage. The exception itself needs an owner because schedules and service accounts change. Give the next shift a way to tell whether a quiet hour means little happened, the rule was changed, or collection failed. Those distinctions also matter when an account review depends on the same events.
Inspect the three records, then use the evidence to answer the question.
1. Incident sample
Rule detected 9 of 12 confirmed incidents.
2. Benign sample
Four of its alerts concerned approved maintenance.
3. Sensor health
One critical log source has stopped sending events.
Apply the reading
What two measures reveal different costs of alert tuning?
Compare your reasoning
Precision measures how often alerts are true and recall measures how many known incidents are detected.