In the previous two blog series articles, we explored exporting objects, alerts and alert actions. We also looked at the concept of converting SPL to KQL. As part of that conversion process their are some special things that will need to be present in order for the queries to run properly on either the Splunk or the Azure Sentinel side.
These items include lookups, source types and indexes.
Lookups (Splunk)
Lookups are an important part of any SIEM/SOAR system. Typically these make up special lists of users, or more commonly Indicators of Compromise such as IP Addresses, Domain Names, etc.
When it comes to Splunk, you can do lookups in a number of different ways. This includes a file upload with can be either a plaintext CSV file, a gzipped CSV file, or a KMZ/KML file.
The maximum file size that can be uploaded through the browser is 500MB.
You can also do script/command based lookups:
So how do lookups translate into Azure Sentinel? There are several ways:
- Log Analytics Table
- Watchlists
- Azure Storage
Picking between the three options depend on the size and use of the actual lookup table. Note, as we will see in a later post, Sentinel doesn’t have a very robust permissions scheme (yet), so they is something to consider as well. In my opinion, this is probably the largest factor when considering moving to Azure Sentinel from another SIEM.
Log Analytics Table
You can use the Data Collector API to bring in Lookups into Azure Sentinel / Log Analytics.
https://docs.microsoft.com/en-us/azure/azure-monitor/logs/data-collector-api
Once imported, they would then show up like this:
Watchlists
Watchlists are the almost identical mapping of a Lookup in Splunk. As you can see here, I imported the exported CSV file programmatically from Splunk into a Watchlist in Azure Sentinel:
Azure Storage
Kusto Query Language (KQL) allows for the runtime creation of a table based on an external file. In this case, we can export the CSV files from Splunk into Azure Storage and then use KQL to bring it into the queries:
Source Types
In addition to lookup data, a vital component is you have to have log data to run your queries on. This log data comes in all kinds of shapes and formats. There are about 150 out of the box Source Types in Splunk.
Source types defined how data is formatted such that the indexers can index it. It is very much an ingest time type of thing.
Indexes
You can consider these like tables in a database. They have a name, rows and columns with type. Splunk stores the data from logs in an “index”. This data is ingested using the source types mentioned above.
In order to determine if a query is going to run, you have to gain access to all the indexes, then export the schema of those indexes. This of course would all go into the tokenizer / compiler and then save the graph of the query for comparison to the “tables” in Azure Sentinel.
As you can imagine, it is not going to be a one to one mapping on the other side. Some things are pretty common, others, not so much.
As we will see in Part VII, Apps have the ability to import new source types, which then in turn create new indexes. As we will see, Apps hold some similarity to “Connectors” in Azure Sentinel.
Ultimately, the goal is to ensure that we have the same Source Type -> Data Source mappings such that the queries are converted properly. In many cases, you’ll need to create a mapping file of the properties on the left side of the equation with the properties on the right side of the migration equation.
Splunk to Sentinel Blog Series
- Part I – SPL to KQL, Exporting Objects
- Part II – Alerts and Alert Actions
- Part III – Lookups, Source Types and Indexes
- Part IV – Searches
- Part V – Dashboards and Reports
- Part VI – Users and Permissions
- Part VII – Apps