From a Log to the Threat Hunting

The proactive search for threats (also called Threat Hunting) is a fundamental process of any cybersecurity strategy that is considered mature. In short, it implies the active search for the attacker within our infrastructure before they can cause any damage.

Within IT environments, tasks and programs are executed, sometimes they generate changes in the environment, or only make use of network resources (for example), all these events are captured by the different applications and reflected as traces in the form of a line of text in a log file, an OS event, a captured network packet, or as we have previously analyzed in the form of a Data Flow (old post) Making an abstraction, we can say that these traces are the minimum unit within threat hunting.

An efficient threat detection strategy will attempt to answer questions about those sources of detection data. Deciding WHAT detection data capture, WHEN capture them, WHEREcapture them, by HOW MUCH time to retain them are some of the pieces that will help us build the best implementation of this threat hunting process.

In this post, we'll explore how we went from a handful of logs and different data types to detection engineering and designing analytics to detect malicious behavior.

About detection

We understand that the detection of the different events in their various formats is the tool that allows us to find a threat. This is why we will emphasize getting the best detections.

Alerting the connection of an unknown device on the network based on the observation of a list of IP addresses (previously registered as valid) seems to be an effective detection method (and it is), but it implies the possibility of generating lots of false positives (alerts that are not of interest). We talk here about Precision of detection.

On the other hand, if we think of signature-based detection, we can be sure that practically all the alerts we receive will be considered of interest and will clearly represent a threat. Although in this way there are lots of other risk situations that will not be included in the signatures, so they will not be detected. We generate lots of false negatives here, so we refer to recall in detection.

We can define these aspects in a diagram taking spam email detection as a reference, with red being spam emails and green being legitimate emails:

Improving the scope of our detections will allow us to “Do not leave malicious behavior out” and the precision will allow us to “Do not mistakenly alert for valid events”.

Additionally, we find in the detection some typical categories that we must highlight before moving on to one of these.

Signature-based detection: In this detection we generally make use of IoCs (Indicators of Compromise) such as malicious IP addresses, file hashes or character strings contained in binaries. Sets of items that have specifically been found in previous incidents.

This detection has excellent precision, but many times we are one bit away from being left with an out-of-date signature.

Anomaly-based detection: This type of detection is based on “defining a normality” in the environment such as a list of applications, lists of equipment, type of network traffic, expected events, and monitoring the differences that exist against this “normal state”. Anything found to be “Out of the Normal” will be considered malicious.

This type of detection can find previously unknown malicious behavior, on the other hand it is not easy to sustain these “normal” patterns, valid behaviors in an environment tend to change (like everything else).

Behavior-based detection: On this type, we will base the detection on the behavior that is generally considered malicious in the particular environment where the detection is made. We can think for example

It doesn't matter if a cybercriminal uses state-of-the-art technology and skills, or if they use malware written in go, python or assembler, their attack will depend 100% on our weaknesses. That gives us an advantage
We cannot therefore advance on this path of detection without going through David Bianco's Pyramid of Pain, where we outline from the base to the tip how the difficulty of replacing an element within an attack increases for the adversary:


From detection to understanding

Generally, in relation to threat hunting, "Looking for a needle in a haystack" comes to mind, but unlike the latter, threat hunting can be improved, with a methodology giving a more effective response, experience shortens its time.

Let's take the following case as a reference:

The “Windows Event Viewer” is run through the Microsoft Management Console (MMC Microsoft Management Console), this implies running in a highly privileged context. Additionally, the "Windows Event Viewer" is a piece of software that tries to run in the first instance based on a registry key setting located in HKCU\Software\Classes\mscfile\shell\open\command when this key is not detected it is executed according to the configuration of HKCR\mscfile\shell\open\command.

You may be asking now:

What happens if within that key, which is available to the user (CURRENT_USER), a value like C:\Users\Documents\TrustMeImNotAMalware.exe is entered?
Let's test it, we try to access the key and modify its value. In our test, the result of this simple test does not seem to have been effective, we even noticed a problem in the registry editor, we tried to run as standard user, even as administrator user. The editor crashed when applying a new value to this key.

A few minutes later and without having found any records about this situation, the SOC team contacted us to inquire about suspicious behavior from the terminal where we were performing the test, an alert referring to a Privilege Escalation tactic:


The question then to start defining the threat hunting process is:

WHY was this behavior being monitored?
The question, although it seems trivial, requires knowing that it's not possible to collect each one of the changes, or events that occur in a group of devices within an Organization. Whether it's a technical issue or a matter of associated resources, something will always be “off the radar”.

On the other hand, the HOW can be understood quickly by doing a little analysis, the modification in that registry key generates some kind of trace, either in the modification of the value itself or in some registry file, that trace is taken (by an EDR solution in this case), and contrasted with a technique, which corresponds to a behavior, that behavior is what we generally call "Tactic", and placed within a context it is the way in which we will respond to the WHY.

We then detect events, but warn about behaviors
MITRE methodology
To shape the entire threat hunting process, we will take a methodology. The MITRE Engenuity team shares a series of stages for this where we will begin with the understanding of the "Malicious Activity" to then decant into specific elements for the execution of specific detections.

Starting from top down on this methodology, in order to detect activity considered malicious within the environment, we must understand that activity. Understanding this threat model then helps us to shape it, for example by using MITRE ATT&CK. Taking the previous case as a reference, we can say:

“An adversary could enhance its execution privileges within the affected environment to cause greater impact”
Moving forward and to achieve a correct analysis and understand what to detect is that we develop hypotheses that can be confirmed from the implementation of a behavior. Continuing with the previous case:

“If the registry key corresponding to the Windows Event Viewer command within the user registry is modified, an adversary is performing an elevation of privilege”
Already below the angle of our methodological V we are very close to the data, what data should we collect to validate the hypotheses? This is where we answer a previous question for our example:

“We evaluate the values of a particular registry key, to detect Privilege Escalation behavior”
From another look and analyzing our V, from left to right we can find on the left the actions of understanding, research and planning around threat hunting, a process often carried out "on paper", here we find behavior descriptions, detection hypotheses and data requirements.

On the right side, on the other hand, we begin to go uphill in the "hunting" process itself. We identify the necessary data, confirm the hypotheses, and detect the threats (or not). It is only here that we will find all the tools, software, solutions, SIEMs, EDRs, IDS, etc

In practice, this methodology is not applied in a linear way, multiple iterations are carried out in different directions, always taking into account the results of each step. The methodology carried out by the hand of an expert could have multiple feedbacks based on new findings for each execution.


There have been some more or less widespread models for hunting threats for years, yet they all share the need to understand what is happening “on the other side” in order to take action “on this side”.

Threat hunting as an active cyber defense implementation allows us to not only detect the adversary, but also generate better methods to move quickly from detection to eradication.

In our next posts we will be exploring the different forms of "hypotheses" for detection, and we will analyze some TTPs (Tactics, Techniques and Procedures) under the lens of this threat hunting methodology.