Detection Engineering

TheoryHard15 minDetection EngineeringSIEM Operation

Writing, Testing and Maintaining Detections

What is it?

This is the craft of shipping a detection you can trust. A detection is written as selection + condition + exceptions in a portable format; it is tested against both true-positive samples and benign data; it is documented with what it catches, the data it needs, and known false-positive sources; and it is maintained, because rules decay as the environment, tooling and adversary change. A rule without tests, documentation and an owner is a liability, not an asset.

Why it matters

Most detection failures in real SOCs are not clever evasions — they are rules that were never tested against normal data, were undocumented so no one could triage their alerts, or silently rotted as the environment changed. The engineering practices here are what separate a detection library that reduces risk from one that quietly stops working while everyone assumes it is fine.

Where you see it

Detections are written in a detection-as-code repository (Sigma or the SIEM's own language), tested in a validation harness against labelled data, documented in the rule itself and a runbook, and reviewed on a schedule. Decay shows up as a rule that once fired now silent, or one whose false positives crept up after a new tool was rolled out.

What normal looks like

A production-grade detection has a clear selection and condition, explicit exceptions, a test proving it fires on the attack and stays silent on benign data, documentation of what and why and how to triage, an owner, and a review date. When the environment changes, someone re-validates and re-tunes it.

What suspicious looks like

A fragile detection was tested only on the attack (never on benign data), has no documentation so its alerts are un-triageable, and no owner or review date so it rots unnoticed. The classic silent failure is a rule that quietly stopped matching after a log-source or field name changed — still 'enabled', catching nothing.

How analysts investigate

Write the rule as selection + condition + explicit exceptions. Test it twice: against a true-positive sample (does it fire?) and against a slice of normal production data (how many false positives?). Document what it catches, the data source it depends on, and how to triage an alert. Assign an owner and a review cadence, and re-validate whenever the environment or the threat changes.

Common beginner mistakes

Testing a rule only on the attack and never on benign data, so its false-positive rate is unknown until it floods production.
Shipping a detection with no documentation, leaving analysts unable to triage its alerts.
Assuming a deployed rule keeps working forever, ignoring decay after log or environment changes.

What you will be able to do

Write a detection as selection + condition + explicit exceptions.
Test it against both a true-positive sample and benign data.
Document it and plan for its maintenance and decay.

Writing the logic is the easy part; making it trustworthy is the job. Three practices do that. Test it twice — on the attack (does it fire?) and on normal data (how noisy is it?). Document it — what it catches, the data it needs, and how to triage an alert — or its alerts are useless to whoever gets them at 3 a.m. And plan for decay — every rule eventually rots as fields, tools and adversaries change, so it needs an owner and a review date.

title: Office app spawns encoded PowerShell
logsource: { product: windows, category: process_creation }
detection:
  selection:
    parent_image|endswith: ['\WINWORD.EXE','\EXCEL.EXE']
    child_image|endswith: '\powershell.exe'
    command_line|contains: '-enc'
  filter_known_admin:              # exception (tuning)
    host: 'ADM-AUTOMATION-01'
  condition: selection and not filter_known_admin
falsepositives: ['approved admin automation on ADM-AUTOMATION-01']
owner: soc-detection-eng   # review: quarterly

A production-grade rule: behaviour selection, an explicit exception, documented false positives, an owner and a review cadence.

Worked example — is this detection ready to ship?

A colleague hands you a rule that targets the right behaviour and fired correctly on the attack sample. They ask you to deploy it. Before reading on, decide: is it ready? The answer: not yet. Firing on the attack proves recall on one sample — it says nothing about precision. You have not tested it against benign production data, so its false-positive rate is unknown; there is no documentation, so an analyst getting the alert cannot triage it; and there is no owner or review date, so when a log field is renamed next quarter it will silently stop matching and no one will notice. 'It fired on the attack' is the beginning of testing, not the end. Test against benign data, document, assign an owner — then ship.

Recap

Write detections as selection + condition + explicit exceptions.
Test twice — on a true positive AND on benign data — before you trust it.
Document, assign an owner, and plan for decay — a rule is never finished.