A DEATHCON Thrunting Workshop Overview Part 4: Baseline Hunting

Detecting normal versus abnormal applications in your environment

If you’re just tuning in now, welcome! This is part 4 of a series on a workshop previously given at DEATHCon 2024. Head over here if you want to start from the beginning.

Welcome back to our DEATHcon Thrunting Workshop series! We are diving into the scenarios provided during the workshop, each covering a different approach to threat hunting, which the PEAK Threat Hunting Framework defines.

Today, we are talking about a baseline scenario, which brings a variety of benefits to both you and your organization. Baseline hunting provides you a snapshot of what is normal in a particular environment and allows you to quickly pinpoint abnormal behavior (either in the current hunt or in future incidents or hunts.)

Why would you want to baseline hunt? Here’s a few cases for conducting a baseline hunt:

When to Conduct a Baseline Hunt:

1. You’re a new hunter in a new company

2. You’re hunting in a new dataset you haven’t hunted in before

3. You’re hunting in a new part of the environment you haven’t hunted before

4. You’re hunting in a newly acquired/merged company

5. You’re revisiting a dataset, network, or environment after major security posture changes

All of these are opportunities for you to baseline what is normal versus not normal. And remember to document all your findings at the end so you can save the next hunter or responder some time.

The scenario we are going to look at today is unauthorized Windows programs. This is based on real challenges hunters and responders see in their business. Unauthorized software can be anything from a Cobalt Strike beacon to World of Warcraft to WinZip, depending on the policies and risk tolerance of your organization. Even if you find nothing unauthorized, it may uncover what software is commonly used throughout your business.

Just like Part 3 we are going to break this scenario up account to the three main PEAK phases - Prepare, Execute, and Act.

---

Prepare Phase - Understanding the Threat

You may be ready to just go diving for unauthorized applications, Ricky Bobby, but let’s first talk about how to prepare for this scenario.

On the surface this is easy, if someone is running a Bitcoin miner, you reimage their computer and talk to their boss right? But what about a cracked version of Microsoft Visio that someone built the latest diagrams for the million dollar client pitch on? What about Thunderbird email client? What about a Pomadoro timer that sits on a desktop to remind someone to take breaks?

When things are really bad, we know what to do with it, but the grey area can make things sticky. Here’s some key things you should sort out before starting:

Key Considerations When Hunting Unauthorized Applications

1. What do your official company policies define as unauthorized applications?

2. Are you hunting malicious applications or also policy violation applications?

3. Do you have sign off from HR to hunt and take action on policy violation applications?

4. How will handle unauthorized applications? Quarantine? Reimage?

5. Is there a known good or trusted software inventory you can refer to?

6. Is that software inventory up to date or has it been updated within the last 6 months?

7. What are your incident procedures if you discover a malicious application?

All of these considerations can help ensure things go smoothly. Knowing what you can and can’t look for, how you will handle findings (because you will have findings) and what is your North Star for approved applications will make this much easier.

What can we expect to see though?

Expected Artifacts with this Hunt

• Unsigned applications

• Outdated applications

• Cracked or pirated applications

• Specialized applications

• Policy violation applications

• Unlicensed applications

• Malicious applications

• Benign applications

What you will find will be diverse and in some cases, significant, and each of these has a different response playbook. If you are in a large organization, it may be wise to chunk this hunt up into different parts of your business, otherwise results may get unwieldily.

Just because we are baseline hunting doesn’t mean we can’t have a hypothesis!

Hypothesis

Attackers are running unauthorized applications, such a RealVNC, on Windows systems to further their foothold throughout the victim organization.

Let’s break it down:

• Technique: Unauthorized applications, like RealVNC

• Target: Windows Systems

• Actions on Objectives: Establishing or spreading a foothold in the victim organization

This hypothesis can be modified based on how many results you see in your baseline hunt or based on historic incident trends you have seen in your organization. If you have an issue with remote management tools, you can focus in on those types of applications, or you could zoom out and focus on all applications running and what is normal versus not.

Everything is wrapped in the Prepare phase, so we are going to dive in to Execute the hunt? Are you ready?

---

Execute Phase - Doing the Hunt

You know I can’t just let you dive in without a bit more preparation right?

Step 1: Initial Data Exploration - Understanding Our Environment

I know, I know, baseline hunts are basically non-stop understanding our environment, but isn’t that the beauty of it? A la Sisyphus we as hunters get to continuously learn about what our business is doing and use that to allow our response teams to get to true bad faster.

Why this step? Before looking for specific anomalies, we need to understand what "normal" looks like in our environment. This helps us:

• Establish baseline software distribution

• Identify common process-to-company relationships

• Discover legitimate business software patterns

This search specifically looks at Windows event logs to figure out the application name and the associated company from the file signature, which allows us to quickly see what major software providers our organization uses.

index=thrunt sourcetype="xmlwineventlog" source="WinEventLog:Microsoft-Windows-Sysmon/Operational" 
| stats count by Company OriginalFileName 
| sort - count

SPL Breakdown

• This query hunts for all applications running by company name and original file name (process name)

• It counts by pairing the company name with the original file name and then sorts by the highest count of the pairs.

Let’s analyze these results together:

Well, obviously someone is heavily invested in Microsoft stock! As we would expect with a Windows dataset. We see a lot of standard Microsoft processes running here (you should be sure to take a look at some of these though) and Google Chrome and VMware. I would refer back to my software inventory, and these are all approved.

But what about the inverse?

Now things are more interesting!

Looking at outliers can usually reveal something fun for you, so the highs and lows often present something for us to chew on.

We see some Microsoft here again, which might deserve some poking, but what is this? RealVNC AND Wireshark? Those definitely deserve a closer look.

We are going to dive into more fun visuals for highlighting anomalies, but before we depart, here’s some analysis notes for this step.

• Look for unknown or suspicious Company names in high-frequency processes

• Identify software that appears to be masquerading as legitimate vendors

• Check for common system processes appearing from unexpected companies

• Note processes with abnormally high or low frequency counts

• Watch for similar process names coming from different companies

Leading to Step 2: Since we understand what software is in our environment, let's visualize how it's distributed across our hosts to better identify anomalies.

Step 2: Process Distribution Analysis

Why this step? Now we'll create a heatmap to visualize process distribution across hosts, helping identify potential anomalies and unauthorized software.

The chart command will help us visualize the results in a table format.

index=thrunt sourcetype="xmlwineventlog" source="WinEventLog:Microsoft-Windows-Sysmon/Operational" 
| chart dc(host) by Company OriginalFileName where count in top20

Let’s analyze these results together:

Ok this part takes a good trackpad or mouse, you’re going to have some scrolling. But I’m a visual learner and the colors really help me find things to look at!

Here I see a lot of deep red, which tells me a lot of hosts are running this particular process. I know some of these are approved, like updater processes for Microsoft or Google, so I am not interested in those. If I wanted to get more precise, I could add exclusions to my search, by company name or process name, but I run the risk something could mask as those names with other suspicious file elements.

Right now I am looking for lighter red sections, or cool spots, which indicate rarity.

And now I see a lot of cool spots I really want to dive in. PolyLogyx, RealVNC, Securax, and Plarium all look very rare, only running on one or two hosts.

Here’s some additional analysis notes for this step":

• Look for "hot spots" indicating widely deployed software

• Note "cold spots" that might indicate rare or unauthorized software

• Check for unexpected variations in deployment patterns

• Identify software that should be consistently deployed but isn't

• Watch for clusters of unusual activity on specific hosts

Leading to Step 3: With our visual baseline established, let's use statistical analysis to precisely identify the rarest software in our environment, which could indicate unauthorized installations.

Step 3: Identifying Rare Software Using Percentile Analysis

Why this step? Instead of using standard deviations, we'll use percentile analysis to identify software that appears on an unusually small number of hosts. This approach is more robust for identifying truly rare software that might indicate unauthorized installations.

index=thrunt sourcetype="xmlwineventlog" source="WinEventLog:Microsoft-Windows-Sysmon/Operational"
| stats dc(host) as host_count by OriginalFileName Company
| eventstats perc10(host_count) as p10
| where host_count <= p10
| sort - host_count

SPL Breakdown:

• First we count distinct hosts running each process

• Then we calculate the 10th percentile of host counts

• We want to show only processes running on fewer hosts than the 10th percentile

• Finally we will sort by host count in descending order (anomaly cheat code here we come!)

Let’s analyze these results together:

So there is quite a bit here but we have the rarest of the rare. I have several leads and more to go after. We know we are interested in Wireshark and RealVNC, but what do the other pages show us?

What is losenord.exe? Is Discord an approved company communication platform? What is crashing and sending data out? Did a Microsoft patch fail to go out? There’s so many questions and leads I have!

Before I dive into each of them, I’m going to refer back to my prepare step for this baseline hunt. We are going to say we have had an issue with remote management software lately. We only have one approved corporate technology but people keep answering tech support scam phone calls and letting others onto their desktop with random programs.

I need to focus in on RealVNC.

I don’t know what is going on with it yet, so I can’t say the presence of the filename == incident. I know it isn’t approved, and I’ve seen our approved software as heavily used throughout the company. So we know our next step!

Before we go, here’s some notes on analyzing for this step:

• Look for software appearing on a very small number of hosts

• Note any unknown or suspicious company names

• Check for legitimate software that should be more widely deployed

• Identify potential shadow IT or unauthorized installations

• Watch for known-malicious software vendors or package names

Leading to Step 4: Now that we've identified potentially suspicious software based on rarity, let's deep dive into one specific finding - RealVNC - which warrants investigation due to its remote access capabilities and limited deployment pattern.

Step 4: Deep Dive into RealVNC Activity

Why investigate RealVNC? RealVNC warrants special attention because:

• Remote access tools are common targets for abuse by attackers

• VNC installations may indicate unauthorized remote access capability

• VNC can be used for data exfiltration or unauthorized control

• Legitimate use should be limited to approved systems and users

index=thrunt sourcetype="xmlwineventlog" source="WinEventLog:Microsoft-Windows-Sysmon/Operational" Company="RealVNC Ltd."
| stats 
    count as execution_count
    values(Image) as vnc_executables
    values(ParentImage) as parent_processes
    values(CommandLine) as command_lines
    values(CurrentDirectory) as working_directories
    by host, User
| sort - execution_count

SPL Breakdown:

• We will track how many times VNC processes executed with stats count

• Let’s collect all VNC executable paths with values(Images)

• We need to identify processes that launched VNC using values(ParentImage)

• We will analyze how VNC was executed with values(CommandLine)

• We can show where VNC was running from with values(CurrentDirectory)

• Finally we want to group findings by host and user for context with our sort variation

Let’s analyze these results together:

This is on two hosts.

There’s a start command.

There’s RealVNC running in C:\temp\.

There’s a license check skip.

Gut feeling? This needs a deep dive, stat.

From my view, two folks in our organization are running an unauthorized remote management software and they need a call or I need to pull their computer logs +/- 15 minutes from RealVNC run to verify why they are running this.

Depending on your preparation, you may be deep diving this run, or it may be time to call your SOC or IR team. Here’s some things you will want to investigate based on what we just saw:

• VNC executables running from temporary directories (C:\temp\)

• Multiple execution modes including service and slave instances

• Installation and configuration activities

• SYSTEM-level access on some hosts

• Various users interacting with VNC software

But wait! There’s more!

• Verify if RealVNC installations are approved and properly licensed

• Check if temporary directory executions align with installation procedures

• Review if SYSTEM-level access is required for identified hosts

• Validate if observed users should have VNC access

• Compare command-line parameters against known-good configurations

---

From Baseline to Beeline

Key takeaway from this hunt

You made it! Hopefully you found a few bad, a few meh, and a few good things running throughout your organization. But now it is time to Act (aka beeline.) We need to put everything we just found into action, because there is always an output from a hunt!

You need to wrap this all up into deliverables that can be shared across the various security, IT, and business groups you work with to make everyone better. So what are we taking away?

What We Caught in Our Net 🕸️

Knowing what is normal versus abnormal in our environment? Proved.

Seeing remote management tools and approved tools running? Proved. yikes

But we also:

• Established normal software distribution patterns across the environment

• Identified statistical anomalies in software deployment

• Discovered potentially unauthorized RealVNC installations

• Built a foundation for future hypothesis-driven hunts

Time to Lock It Down 🔒

In the Short Term (Do This ASAP):

1. Figure out why those two folks are running RealVNC - those hosts and users need some love

2. Identify if a phishing email needs to get pulled or a malicious drive by website needs to be reported that led to the RealVNC installs

3. Craft some user education around the attack and remote management applications

Future Hunt Ideas: 🎯

• Track software installation patterns over time

• Investigate network connections from rare software

• Analyze parent-child process relationships for anomalies

• Cross-reference with other data sources (e.g., network traffic, authentication logs)

Process Improvements: 🧹

• Update software approval workflows

• Enhance asset inventory processes

• Refine software deployment procedures

• Strengthen remote access tool governance

Knowledge Sharing and Documentation: 📝

• Document baseline software patterns for future reference

• Share findings with security and IT teams

• Update threat hunting playbooks with new techniques

• Contribute insights to the threat hunting community

Detection Engineering: 🛠️

• Create alerts for new software installations outside approved paths

• Develop detections for unauthorized remote access tools

• Implement monitoring for statistical deviations in software deployment

• Set up alerts for suspicious process execution patterns

Metrics and Impact: 📈

• Number of unauthorized software instances discovered

• New detections created from findings

• Process improvements implemented

• Coverage gaps identified and addressed

In the Long Term:

• Figure out how to block unauthorized applications

• Review if SYSTEM-level access is required for all hosts

• Lock down temporary directory access

Remember: Baseline hunting is iterative. Use these findings to inform future hunts and continuously improve your security posture.

Stay curious and happy thrunting!

---

Looking for other hunt ideas?

That’s exactly why we built HEARTH, the Hunting Exchange and Research Threat Hub. It’s a community-driven repository where hunters can share ideas, experiment with new approaches, and contribute to a growing collection of threat hunting ideas. Every contribution gets a unique hunt number and a sticker!

🔥 Want to submit an idea or see what others are hunting? Check out HEARTH.