Hunting at Scale: Stack Counting and Least-Frequency Analysis
What is it?
Hunting at scale is finding the needle in millions of events without a signature to match on. The core techniques are aggregation-based: stack counting groups events by a field and counts them, so you read the distribution instead of individual lines; least-frequency-of-occurrence analysis then focuses on the long tail — the rarest combinations — because on a well-run network malicious activity is usually unusual, and unusual is rare.
Why it matters
You cannot read millions of process events by eye, and you have no signature for an unknown threat — so you need a way to make rare, interesting things rise to the top on their own. Stack counting and least-frequency analysis are that way: they turn an impossible read-everything problem into a short list of outliers a human can actually examine. This is the workhorse technique of applied hunting.
Where you see it
Stack counting is run in the SIEM or a notebook: group process-creation by parent-child pair and count; group outbound connections by destination and count; group logons by account and source. The long tail — the entries with a count of one or two — is where you look. It works on any high-volume field where 'common' means 'normal'.
What normal looks like
In a stack-counted distribution, the head is the everyday: explorer spawning browsers tens of thousands of times, services spawning svchost thousands of times. These high counts are the baseline — common, repeated, explainable activity that forms the bulk of the data.
What suspicious looks like
The tail is where hunting lives: a parent-child pair seen once among millions, a destination one host talked to that no other did, an account that logged on from a source used nowhere else. Rarity does not prove malice — a rare admin action is still rare — but on a well-baselined network, the long tail is where the unknown intrusion hides, and each rare entry earns a look.
How analysts investigate
Choose a field where common means normal (parent-child pairs, destinations, logon sources), aggregate and count across the whole dataset, sort ascending, and examine the long tail. For each rare entry, apply context — is a document application spawning a scripting host, is this destination unknown — to separate the rare-but-benign from the rare-and-suspicious. Then pivot on the suspicious ones like any lead.
Common beginner mistakes
- Trying to read raw events one by one instead of aggregating and counting.
- Treating rarity as proof of malice — a rare legitimate action is still legitimate.
- Stacking on a field where 'common' is not 'normal', so the long tail is meaningless.
What you will be able to do
- Use stack counting to read a distribution instead of raw events.
- Apply least-frequency analysis to surface the long-tail outliers.
- Separate rare-but-benign from rare-and-suspicious with context.
You have a hypothesis and a data source with millions of rows. You cannot read them, and you have no signature. The trick is to stop looking at events and start looking at the distribution: group by a field where 'common' means 'normal', count, and the picture inverts — the enormous benign majority collapses into a few high-count rows at the head, and the interesting anomalies fall out as the rare long tail.
| Parent → child | Count | Read |
|---|---|---|
| explorer.exe → chrome.exe | 48,120 | Head — everyday baseline |
| services.exe → svchost.exe | 9,455 | Head — normal |
| WINWORD.EXE → mshta.exe | 1 | Long tail — look here |
Worked example — read the stack
The table above stacks millions of process-creation events by parent-child pair. Before reading on, decide which row you investigate and why. The answer: WINWORD.EXE → mshta.exe, count 1. The two high-count rows are the baseline — browsers and service hosts launched constantly, exactly what a healthy network does all day. The single occurrence at the tail is the outlier: an Office application spawning a scripting host, once, among millions. Rarity alone would only flag it; context convicts it — a document app has no legitimate reason to launch mshta.exe. Note what you did: you never read a raw event, yet the needle surfaced itself. That is hunting at scale.
Recap
- Stack counting reads the distribution, not raw events — the baseline collapses into the head.
- Least-frequency analysis surfaces the long tail, where the unknown intrusion hides.
- Rarity flags; context convicts — a rare legitimate action is still legitimate.
Sign in to save your progress on the server.