Practical Guide to ESM Active Lists
Active Lists are ESM resources that store event data/fields (not entire events) for a definite or indefinite period of time. They can store defined event fields (event-based active lists) or data extracted (and maybe manipulated, i.e. converting destination usernames to uppercase) from event fields (field-based active lists).
This practical guide is intended to provide an overall overview and hands-on on how to use this valuable resource within ESM.
This guide covers:
- Event-based vs Fields-based
- Key Fields
- Multi-mapped vs Non multi-mapped
What this guide is…
- A document to be used as a basic reference to help you effectively build your ESM content
- A quick start guide to help you get up to speed in ESM active lists authoring
What this guide is not…
- A written-in-stone guide. As every environment is different, different conditions or assumptions apply to them
- A replacement for ArcSight ESM Training
- A replacement for the ESM Console User guide or any other ArcSight official guide
When creating an active list the first and most important thing to consider, other than to include the customer resource column, is to think about what the active list is going to be used for. This will determine the type of list to be created and the fields (and their types -field-based list) to be used. Once the list has been created you cannot modify its schema (fields and types) but if a change needs to be made then a new active list has to be created. The active list should hold the right data as other resources will be using it.
Active lists can be populated either by a rule action when a rule is triggered (either standard or lightweight rule) or manually through the ESM Console -by hand or importing a csv file. Then this data can be queried and used by other resources as part of the correlation evaluation process -rules, data monitors. Or as part of the Monitoring, Reporting and Incident Analysis phases of the event lifecycle by using queries, query viewers and reports.
In this example we have an active list that is being populated by a lightweight rule and then its data is being queried by a second standard rule, and also through a query, both a report and a query viewer are accessing its data.
Event-based vs Field-based
First thing to consider is the active list type to be created. There are 2 types:
- Event-based active lists have an explicit event field associated with every field in the active list. These are "fixed" active lists. If source address event field is included as a field in the list, this field will always be referenced as a source address (and maybe you want to match this IP address later as destination or device address in a different rule) . You cannot use variable functions, such as type-conversion functions, in this type of list. Also cumulative fields (sum,max,min) cannot be used with event-based active lists.
- Fields-based active lists have generic fields with only a data type specified. Any event field (or part of it) or different external data (like Threat Intelligence data) can be used to populate the active list fields as long as the data type is matched (you can use type-conversion variables to convert an IP to a string for instance). This type of list is more flexible as, for instance, you can have a field called IPAddress which is populated by the source Address field of event A, and later it can be matched to Destination/Device/Source Address fields of event B -as being evaluated by another rule. You can also use other variables (such as Concatenate,ToUpper,Add,Round) to store the data that you actually need within the active list.
You can use Key fields with both types of active lists.
According to the ArcSight ESM Console User guide, there are 2 best practices for them:
- For a time-partitioned active list, key fields must be a Date field and a string field
- Cumulative (COUNT for instance) fields cannot be used as key fields
Keys allow to map one single key to more than one value (using multi-mapping). Rules can look up value fields for given key field values.
If you are going to do anything other than check an entry to see if it is in an active list, you will need to access the data through the GetActiveListValue function, which means you need to define key fields. Key fields are also required when you want multi-mapped entries, case insensitive columns, etc. Choose your key fields wisely.
Key field selection should satisfy the following conditions:
- Use the minimum number of key fields possible, but don’t be afraid to have multiple key fields.
- The key field(s) selected should uniquely identify an entry.
- The event fields from which the data is taken should never be null.
- When using IP address fields as a key field, always add the zone field as a key field, as well.
- Don’t forget to use the customer field as a key field.
Example 1: You want to track the version status of connectors.
You can use an AL with the following columns:
- Customer (key)
- ConnectorID (key)
The Customer field is a key field and can be used to select connectors belonging to a given customer for queries on the AL. The ConnectorID is a key field because it is an ArcSight resource ID, which is guaranteed to be unique in the system. The rest of the fields are value fields, and with respect to connectors, should be fairly stable, although things can change, especially the ConnectorURI and the ConnectorVersion.
Example 2: You want to track the version status of systems.
You can use an AL with the following column set (column set 1):
- Customer (key)
- SystemZone (key)
- SystemAddress (key)
If your systems use DHCP, you might instead use this column set (column set 2):
- Customer (key)
- SystemZone (key)
- SystemHostName (key)
Multi-mapped vs Non multi-mapped
Multi-mapped active lists are used to allow multiple instances of key (fields) pairings, allowing to map them to multiple values. For instance an internal host (defined by zone and ip address - key pair) to have multiple users, files, software installed or versions (value fields). A user (customer and destination user name Key pair, for instance) can have logins to different systems (value field).
In this example, we are going to review the behavior of both Multi-mapped and non multi-mapped active lists in the same scenario by creating active lists to track how many hosts the same malware has infected in a defined period of time (defined by the active list TTL). The fields that will be used are:
- zone: Destination Zone field (Key)
- malware: Name field (Key)
- client: Destination HostName field
Tip: While this is one way to solve this Use Case, it might not be the only one.
To configure a no multimapped active list we simply left unchecked the Allow multi-mappings setting:
Multimapped active list: We use the exact same fields but we check the Allow multi-mappings setting.
We send 6 same-malware events for 3 different hosts (2 events each) to the ESM Server:
If we send a non multimapped active list we get a single entry for all events, and the count is 6:
This multimapped active list can be used:
- In rules, to trigger when certain number of hosts (or similar use cases) have been infected by the same (or different if we change some fields) malware in the same ArcSight Zone.
- In Reports
- In a Dashboard, to show statistics regarding Malware Infections:
An active list query will count the number of hosts, using the COUNT function for the clients field, to actually count the number of hosts per key pairing - zone and malware.