Practical Guide to ESM Filters - Part 1
Filters are a set of conditions (by using Boolean operators) that focus on particular event attributes, reducing the number of events that are processed by the ESM Server.
Filters are applied at 2 different levels: ESM Server and Connectors. Within the ESM Server, the same filter resource can be used by different resources such as rules, queries, reports, query viewers, trends, data monitors, and active channels.
This practical guide is intended to provide an overall overview and hands-on on how to create this valuable resource within ESM, providing some hints and tips along the way.
This first part of the guide covers:
- Creating a basic filter
- Filter operators
- Evaluating conditions
- Reusing and nesting filters
- Alternatives to piled OR'd conditions
- NULL conditions
What this guide is…
- A document to be used as a basic reference to help you effectively build your ESM content
- A quick start guide to help you get up to speed in ESM filters authoring
What this guide is not…
- A written-in-stone guide. As every environment is different, different conditions or assumptions apply to them
- A replacement for ArcSight ESM Training
- A replacement for the ESM Console User guide or any other ArcSight official guide
Creating a basic filter
1. Within the ESM Console follow these steps to create a basic filter: In the Navigator panel in the left part of the console, on the drop-down list select Filters.
2. Right-click the folder under you want to create you filter and from the submenu select New Filter.
Tip: As a general practice resources (Filters in this case) should not be created within the User’s personal folder (Admin in this case).
3. The Filter Editor Dialog box appears in the Inspect-Edit Panel. In the Attributes tab the name and description will be set.
Tip: Rules can also contain conditions without referring to a filter resource. For specific differences refer to ESM Console User guide.
4. Having highlighted the Event node, click the appropriate operator (AND,OR,NOT)
5. There are 2 ways to add conditions to a filter:
a) Right-click the operator and from the sub-menu select New Condition and then select the corresponding event field.
After selection is made type the proper value and click OK.
If a change inthe operator has to be made, double click the operator (“=” in this case), and from the submenu will appear select the appropriate operator.
Tip: Not all operators are available for all event field types.
b) In the CCE (Common Conditions Editor) look for the corresponding event field and type the proper value in the Condition Tab.
Tip: After entering a value, another event field of the same type will be added right below allowing you to enter another event field of the same type.
Tip: If when selecting the event fields the desired field does not appear in the CCE, we can click on the CCE drop-down list and click the Clear button, this will make all event fields visible.
These are the main (Boolean) operators used within ESM Filters:
- AND operator: Evaluates ALL conditions to return a positive result
- OR operator: Satisfied if at least one condition is matched
- != (Not Equals) operator: Excludes one or more know values
Operators can be changed by right-clicking the operator and from the sub-menu select Change Operator and then the desired new operator.
Tip: Another faster way to change the desired operator is double-clicking on it.
Conditions are evaluated in order of appearance from top to the bottom. They are evaluated using what is called "Short-Circuit Evaluation", which means that in a simple AND condition, if the first element is false, it doesn’t matter what the second element is, both must be true for the AND condition to be true. A similar circumstance applies to OR condition, if the first element is true, it doesn’t matter what the second element is, because only one element needs to be true for the OR condition to be true.
When evaluating conditions some guidelines are recommended to enhance the ESM correlation engine performance:
- AND Operator:
Conditions should be ordered from the most to the least restrictive one. As all conditions have to be matched for the operator to return true; making the most restrictive condition the first one in the operator eliminates the need for further condition evaluation if such condition is not met; and also restricts the number of events to be further evaluated.
- OR Operator:
Conditions should be order from the least to the most restrictive one. As the OR operator returns TRUE when ANY condition is matched, setting the least restrictive one as the first one in the operator eliminates the need of evaluating more conditions.
· Integer comparison, null test
· String equals
· String comparison (Start with, Ends with)
· Conditions on variables (cheap functions)
· InActiveList conditions
· Matches operator (depends on regex)
· Conditions on variables (expensive functions)
· Asset Conditions
* Drag and Drop is the easiest way to change conditions order.
Here we have an example of how we can re-order a set of conditions within a given filter:
We added conditions as follow:
We re-order as follows, based on XYZ assumptions:
- DecID is the most restrictive or “unique” condition so we move it to the top (AND operator). We greatly reduce the events that will be further evaluatedThen we want to make sure the event comes from our Corporate firewall
- We also believe that it is more likely to match more events going to our devops.lab.org domain (OR operator)
Tip: When you modify an exisiting filter, you can copy all the conditions in a new filter and then modify the existing one. This will preserve original content in case you make a mistake.
As we explained before, every ESM environment is different as well as the events data flowing to it, so order your conditions in a way that will best suit your environment!
Not all variables have the same weight in terms of processing time:
o Fastest: Arithmetic Functions (Add,Substract) and Simple String (ToUpperCase)
o Medium: Conditional evaluation, Get Active/Session ListValue
o Expensive: GetGroupsofAsset, FormatGroupsOfAsset
o Most Expensive: Chains of expensive variables, GetListValue on partially cached AL
Reusing and nesting filters
Reusing and nesting filters is a common practice and helps keep things organized (but be careful of not over-nesting your filters in an infinite loop):
1) On the original filter right-click and cut the conditions that will be added to the new filter.
2) On the new filter right-click the Event node and select Paste from the sub-menu
Conditions will be copied
Save the filter.
3) On the original filter click on the Filters option up in the window and from the Filter Selector window select the recently created filter.
4) Put the nested Filter in the correct order by using the Drag-and-Drop feature.
5) More filters can be created/nested. In this example other filter is created and nested for the Corporate Firewall events.
Alternatives to piled-up OR’d conditions
When creating a filter and start adding conditions to an OR operator we may find ourselves in a situation when we have already added too many conditions with the same event field looking for different possible values:
There are 2 ways to work this situation in a more efficient way:
a) Changing the operator to In and then add the list of possible values
b) Create an active list with the fields/values required and then use an InActiveList condition that references to such active list
NULL/NOT NULL conditions are commonly used in ESM to validate if a value exists for a given event field. This is NOT the correct way to implement it:
1) Double-click the condition you want to change, then click the = operator and change it to Is.
2) Select either NOT NULL or NULL from the drop-down list.