Detection Validation & Tuning

TheoryHard13 minDetection EngineeringSIEM Operation

Reading a Detection for False Positives and False Negatives

What is it?

Validating a detection is reading its logic to predict two failures: false positives (it fires on benign activity) and false negatives (the real technique slips past it). Precision is how many alerts are true; recall is how much of the technique it actually catches.

Why it matters

A detection is only as good as its two error rates. Low precision buries analysts in noise until they mute the rule; low recall lets the attacker walk past a rule everyone believes is protecting them. A senior analyst predicts both before trusting a detection.

Where you see it

In the rule's own logic (its selection and conditions), in a backtest against historical data (how many hits, how many benign), and in the alert queue where a noisy rule announces its own low precision.

What normal looks like

A healthy detection targets an invariant of the technique — something the attacker must do — with tight conditions, so it fires rarely and almost always on the real behaviour.

What suspicious looks like

A rule keyed on an easily-changed indicator (a specific filename or IP) has high false negatives — the attacker changes one field and vanishes. A rule with a broad, benign-heavy condition has high false positives.

How analysts investigate

Read what the rule keys on: is it an invariant of the technique or a swappable indicator (that predicts false negatives)? Then backtest against benign history: how often would it have fired on normal activity (that measures false positives)? Judge precision and recall together.

Common beginner mistakes

Judging a detection only by its false positives (noise) and never asking what real attacks it would miss.
Keying a rule on an indicator the attacker controls (filename, hash, IP), giving it high recall on paper but none in reality.

Two errors, judged together

                     detection fires?
                     YES              NO
  real attack   │  TRUE POSITIVE   │  FALSE NEGATIVE  ◀ recall gap (misses)
  benign        │  FALSE POSITIVE  │  TRUE NEGATIVE
                     ▲ precision gap (noise)
  keyed on an invariant of the technique  ─▶  high recall (few misses)
  keyed on a swappable indicator (IP/name) ─▶  low recall (attacker edits one field)

Precision is about false positives (noise); recall is about false negatives (misses). A good rule targets a technique invariant, not a swappable indicator.

Worked example. Rule A alerts when the command line contains the exact string 'Invoke-Mimikatz'. Precision looks perfect (it never fires on benign traffic) — but recall is near zero: the attacker renames the function and walks past. Rule B alerts when a process reads LSASS memory with a suspicious access mask from a non-security tool — an invariant of credential dumping. B fires on the technique regardless of names, so its recall is high; you then tune its precision by excluding the two legitimate tools that also read LSASS.

Quick check

A detection keyed on a specific malware hash has never produced a false positive in a year. What is the most likely problem?

A quick self-check — it doesn't affect your XP or progress.