Do you have feedback on our new interface?
Do you have feedback on our new interface? Let us know HERE
Highlighted
Micro Focus Expert
Micro Focus Expert
978 views

What custom integrations or scripts would you like to see? Community feedback appreciated!

The upcoming weeks i will be releasing a lot of new things to help people use and understand the API's available for ArcSight ESM and ArcSight Logger, after that i would like to create some new custom functionality and scripts that the community can enjoy, the only issue is that it would not really be efficient to create something that no one would want to use.

Please let me know if there is any custom work that you would love to see when it comes to interaction with these two products. All will be released on github in case people would like to make their own changes and tweaks, together with proper documentation and commented code for learning purposes.

Any type of request is appreciated, and the only requirement is that it is not related to a product that is behind a license wall as i would not be able to test it or access API documentation to the product.

A few examples of requests that i can think off:

1. Slack or messaging notifications. Being able to create a rule action that notifies your slack channel when an alert happens.

2. Open Source threat intelligence framework, scripts that retrieve threat intelligence sources from a large amount of open source feeds, being able to choose which ones you want to use, and feeding it to ESM through a syslog connector.

3. Logger interaction script. Something that you can manually run to for example return the results of a query in CLI. Could also be scheduled to create scheduled exports of certain queries.

UPDATE:

First version of Request Tracker integration with ESM has been released: 

https://community.softwaregrp.com/t5/ArcSight-User-Discussions/Unofficial-ESM-Request-Tracker-RT-Ticketing-Integration-released/td-p/1672538

First version of the ArcSight Logger API documentation + examples:

https://community.softwaregrp.com/t5/ArcSight-User-Discussions/Unofficial-API-documentation-and-examples-Part-1-3-ArcSight/td-p/1674083

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
18 Replies
Micro Focus Expert
Micro Focus Expert

Re: What custom integrations or scripts would you like to see? Community feedback appreciated!

One last run to see if anyone more is interested! Got some PM's about it already, though better to put it in here.

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
Knowledge Partner
Knowledge Partner

Re: What custom integrations or scripts would you like to see? Community feedback appreciated!

@Marius,  Are you going to give any examples for ESM API? That would be great.

For Logger API, I think it's possible to do a simple device monitoring. Here is my scenario: pull device Address and device Hostname from a search( we can do this by using dedup operator) and compare it to a list in a file(csv or something like that). After comparing, export missing or extra device informaiton to a file or sent an email.
There are so many customers just having logger and not capable of monitor the status of devices. I believe this would be awesome.

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: What custom integrations or scripts would you like to see? Community feedback appreciated!

@mr_ergene Yeah absolutely, any integration examples will be fully available and documented for learning purposes towards the community.

If you mean the examples i will be giving out, then yes, it will cover both Logger and ESM.

Logger is quite small, so i will cover all API calls there, on ESM i will deliver 2 versions, one for getting started with all the most used API calls, and then a bit later one with all API calls.

The last one covering all API calls takes a bit extra time, as there is certain API calls i have to keep out as they are dangerous, while i like to test all of them before releasing them (current count of API calls is 2000, though many are redundant of eachother, which is why i am stripping out certain ones).

All in a easy exportable format, that you can import with this tool that i use for all my API testing before adding them in whichever script or software i am developing with: https://www.getpostman.com/apps.

I will provide some quick examples on how to import and get started, but the rest is up to the developer :) Though ofc any API specific questions can be posted, and i normally pick them up quite quickly.

It's all a part of a followup after the presentation i had at the Cyber Security Summit in Washington, no way to direct link it, but was 9:50 on Thursday: http://www.cvent.com/events/micro-focus-cybersecurity-summit-2018/agenda-f2b113e2a7234e0b8680e70404728f68.aspx

For your logger example i am trying to understand what you meant.

You want to run a search, that returns all unique device names/addresses from that search, and IF any of the devices from your local list is not in the search results, then create a notification by email etc?

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
Knowledge Partner
Knowledge Partner

Re: What custom integrations or scripts would you like to see? Community feedback appreciated!

Thanks for the details :)

For the logger example, yes I meant that. If there is an alternative way for device status monitoring using logger, that would be also OK.

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: What custom integrations or scripts would you like to see? Community feedback appreciated!

Hmm, when enabling device health monitoring on a connector, i am quite sure it can also send this to the Logger, so i could just simulate the ESM implementation of that in a few lines of code i think. Let me add it to the list.

I am always open for some more challenging approaches as well, feel free to suggest anything! :)

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
Knowledge Partner
Knowledge Partner

Re: What custom integrations or scripts would you like to see? Community feedback appreciated!

@Marius,

Device monitoring doesn't work properly on smart connectors even the aggregation for internal events is disabled. That's why I mentioned about running a query with dedup operator on deviceIP and deviceHostname :)

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: What custom integrations or scripts would you like to see? Community feedback appreciated!

Hmm, so connector monitoring would be a piece of cake, as Internal Event Storage includes minute by minute EPS count per Connector, but drilling it down to how many EPS per device is not really available at this point that i could find.

I also didn't find any Agent:043 events, as i was hoping it at least sent a copy of that to the Logger (might be missing a destination configuration parameter for that though.

The issue is really if you have 10k devices, these types of monitoring solutions would be limited to connector level, as it would be too much to do statistics on that many devices in each search (also because 10k is the max limit for one API call, though you can bypass that in certain ways).

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
sharan Bhat Respected Contributor.
Respected Contributor.

Re: What custom integrations or scripts would you like to see? Community feedback appreciated!

Hello Marius,

1.  Adding to "point 3" from your post. A script that could pull the dashboard logs/Job execution status of the Arcsight logger via API. My idea is to automate the healthcheck of the logger software without manual intervention. I am aware that reports and searches can be pulled using the REST/SOAP API but not sure about this.

2. This may seem farfetched. But what if the smart connectors had the option to fix log stoppages itself without manual intervention? Auto restart for starters(not from ArcMC). Or even a notification with a detailed analysis based on the error logs along with suggestions.

regards

Sharan Bhat

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: What custom integrations or scripts would you like to see? Community feedback appreciated!

Hey @sharan Bhat

The Logger API is only limited to Search and Reports. Reports being SOAP only for some reason..

It is not all lost though, all depending on what you want to monitor. It would be quite straigth forward to create monitoring for these events, as they are internal events available from searches:

CPU:
/Monitor/CPU/Usage cpu:100

Disk:
/Monitor/Disk/Read disk:102 
/Monitor/Disk/Write disk:103

EPS:
/Monitor/Receiver/EPS/All eps:100
/Monitor/Receiver/EPS/Individual eps:102
/Monitor/Forwarder/EPS/All eps:101
/Monitor/Forwarder/EPS/Individual eps:103

Memory:
/Monitor/Memory/Usage/Platform memory:100

Network:
/Monitor/Network/Usage/In network:100
/Monitor/Network/Usage/Out network:101

Search:
/Monitor/Search/Performed search:100

Storage Group:
/Monitor/StorageGroup/Space/Used

Would these be sufficient? For example i could enable the user to fill in a template, of min and max values that any of these values is supposed to have, like:

Disk 10%-80%, CPU 0-90% etc etc. Then it will just notify you if any of the values are outside of your "boundaries". Sounds useful?

For connectors, this is normally done through basic system administration scripts, and outside of the product itself. Normally what you could do here, is have a cronjob that runs every 10-15 minutes, checks that logs are coming in (by checking the last time the queue file was edited, then also check if cache is building up and that the process is running.

Then if queue file is old but process is up, it should notify, if cache is building up it can notify, and restart process if down.

The only issue here, is if you want to automate this process, like removing big cache file, you are essentially losing data, so automatic intervention would not always be such a good idea.

Do you have any specific examples on things that happens, and what you want to automate in response on connectors?

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
Frenjd Valued Contributor.
Valued Contributor.

Re: What custom integrations or scripts would you like to see? Community feedback appreciated!

@Marius

A couple of things I am about to start looking into for a customer, I am not sure if they are sutiable for your integrations but here they are:-

Right click funtionality in ESM to create a ticket in Request Tracker Ticketing system.

Basically an event comes in to an Active Channel and the customer wants the Analyst to Right Click on that event and have it generate a ticket in RT

https://bestpractical.com/request-tracker/

Also a quick and easy way to monitor current cache sizes of all connectors in the event of ESM not being available.  I was going to try to do this in stages with the first stage being to collate current cache sizes of all connectors.

Ideal features are:-

  • Current cache size
  • available cache remaining
  • free disk space remaining
  • based on the above, a prediction on how long the connector can continue to cache before events will be dropped
  • connectors that are dropping events, if possible include a count of dropped events.

How much of the above is poossible I do not know but having just suffered from an incident of ESM being down due to a hardware failure this is quite a hot topic.

 

 

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: What custom integrations or scripts would you like to see? Community feedback appreciated!

@Frenjd

For RT i can create a simple script that creates a ticket in RT sure, as the API documentation is available.

For connectors, it seems that quite a few are requesting this, and i am a bit suprised as to why, there is a few things:

ArcMC is mandatory in later versions of ADP, utilized to monitor license usage etc. ArcMC is always the best place to monitor and manage your connectors, including cache size etc. Using a manual script to do this is possible for some of the usercases, but they really should implement ArcMC instead.

Predictions and statistics requires access to the history of the growth, since ArcMC does not have an API, it would have to retrieve it from ESM, or keep it's own history. 

From the requests i got in this thread, what i could see is:

@mr_ergene

Logger API script for device monitoring, not really possible i am afraid, as Logger only stores connector monitoring data. ESM API examples should be coming soon at some point though.

If i was to do device monitoring on logger, it would need to be done through the report API, with heavy queries that would break if you got lots of devices + large amounts of data.

@sharan Bhat

1. I checked the overview of all internal events that generates on the logger, and there is no job status or report status internal events that i can access unfortunately :( The API itself only focuses on search.

2. Detailed automatic log analysis is a bit out of scope, those really should be integrated into your central logging solution instead. A script could restart connectors etc, but i wonder how much time you would save.

@Frenjd

1. RT Ticket creation should be just fine, should it be able to click on a event, and update an existing ticket with new info as well?

2. See answer at the top.

 

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
Frenjd Valued Contributor.
Valued Contributor.

Re: What custom integrations or scripts would you like to see? Community feedback appreciated!

@Marius

Being able to create a ticket in RT from an event would be an excellent feature.  If you could make it even better by being able to click on an event and update an existing ticket, it would be really great. This would really be beneficial and it would really add some value to ESM.

I am not sure if it is possible but if you could reuse the same code have a feature to create a ticket by an action in a rule it would be the cherry on the cake.

For the connectors, I see your point with regards to ArcMC and I do totally agree.  However we have a multi-tenant setup here and there are physically seperated networks that will never be joined for a number of reasons.  Therefore we have a number of ESM installations.  Of those installations only one has ArcMC, therefore in this paticular case it would be a huge benefit to have a script available that can provide this information when ESM is not available.

With regards to Predictions, I was thinking of this being short term.  A scenario being ESM is unavailable and the connector starts to cache.  The script would read various paramters from that point forward including cache size, number of cached events, size of remaining cache, free disk space etc.

Say a connector has started to cache and after 2 hours it has 24,000,000 events in cache.  4 hours later that figure is 72,000,000 events.  Currently the event rate is increasing at 12,000,000 events per hour or an EPS of 3,333 .  Over a 24 hour period the figures decrease per  hour as there is less event generation out of hours. 


After 24 hours of events being cached, we can see that there are 110,000,000 events in the cache.  Over the 24 hour period that gives us an average EPS of 1,273.

Based on those numbers we could roughly predict when events would start to be dropped based on comparing the resources consumed after 24 hours of caching and using the remaining resource availability.  If one day has exhausted 25% of the available resources we can predict that we have 3 days of caching remaining before we start to drop events.

I know the figures can fluctuate and it would be rough calculations but during an ESM outage the customer would find it very reassuring for me to give them a ball park figure of how long we can continue to cache events before they start to get dropped.

I know there are a number of HA options available but this use case is one where those options are not implemented.  And this is a real case based on a live customer outage.

Thanks 

 

 

0 Likes
Micro Focus Expert
Micro Focus Expert

Re: What custom integrations or scripts would you like to see? Community feedback appreciated!

Hey @Frenjd

Just a quick update, i am working on the RT integration, was unfortunately hard to find some time, but i finally got that earlier this week, and i finished all the functionality, it can now:

1. Create ticket from rule

2. Create ticket from user interaction (right click)

3. Update existing ticket with new events with user interaction (right click, and fill in ticket id to update)

It has a template system, very easy to understand, you fill in the values of the fields in RequestTracker on the left, and the name of the fields you want to map from ArcSIght to the right, and you can add ofcourse just normal text as well.

Example:

[template1 - malware]
id=ticket/new
Queue=General
Requestor=root@localhost
Priority=arcsight.priority
Subject=arcsight.name
Source=sourceAddress
Destination=destinationAddress

[template2 - bad url]
id=ticket/new
Queue=General
Requestor=root@localhost
Priority=arcsight.priority
Subject=arcsight.name
Source=sourceAddress
Destination=destinationAddress
URL=requestURL

This enables you to right click, and have several "create ticket" options, based on what fields are available for your type of event, this also enables you to use the service on any RT installation, no matter how many custom fields the user might have.

Everything is tested and working, just commenting the code and writing a installation guide, after that i will release it, most likely this weekend or start of next week :)

Have a good evening!

 

-----------------------------------------------------------------------------------------
All topics and replies made is based on my personal opinion, viewpoint and experience, it does not represent the viewpoints of MicroFocus.
All replies is based on best effort, and can not be taken as official support replies.
//Marius
0 Likes
Frenjd Valued Contributor.
Valued Contributor.

Re: What custom integrations or scripts would you like to see? Community feedback appreciated!

@Marius

 

That sounds absolutey brilliant, I am really looking forward to seeing it :-)

Thanks, hope you have a great evening

0 Likes
The opinions expressed above are the personal opinions of the authors, not of Micro Focus. By using this site, you accept the Terms of Use and Rules of Participation. Certain versions of content ("Material") accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.