ArcSight Discussions
cancel

Inquiry about ArcMC License Usage data calculation

yhann1 Trusted Contributor..
Trusted Contributor..

Inquiry about ArcMC License Usage data calculation

Dear awesome community:

I'm wondering how ArcMC obtained the following GB/Day figure from the Loggers which are being managed by ADP License Server:

img1ArcMC1.jpgAs above shows, we are currently using average 120++ GB/Day out of maximum 300 GB per day. There are 2 Loggers managing by this ArcMC.

However, if we manually perform the Data Volume report from Logger OR the Logger Consumption report from ArcMC, their GB/Day reading is different from the img1 dashboard reading! Refer below:

img2 (from ArcMC Logger Consumption Report)ArcMC2.jpg

img3 (from actual Loggers Data Volume view)ArcMC3.jpg

As we can see from above two images, both Logger consumption for today adds up a total of 25GB. It is a massive difference from what we see from ArcMC dashboard of average 120++ GB/day.

We will have two more new Loggers with massive daily Data Volume usage of average 100GB or more. I'm particularly worried about how ArcMC will calculate both of these new Loggers license usage data should we connect them to ArcSight. If it exceeded the 300GB Licensed GB per day threshold, will all managed Loggers functionality be restricted? 

* Isn't License Usage data reading referring to actual Logger Data Volume? From the help menu of ArcMC:ArcMC4.jpg

ArcMC 2.7.1.2065.0

Logger 6.3.1.7874.0

10 REPLIES
Micro Focus Expert Marijo Mandic
Micro Focus Expert

Re: Inquiry about ArcMC License Usage data calculation

Hello,

I would advise to create Service Request that this is investigated as many things need to be checked to confirm the accuracy of all parameters.

Regards,

Marijo

hatemware  Super Contributor..
 Super Contributor..

Re: Inquiry about ArcMC License Usage data calculation

Hello yhann1

I have the same issue here so did you manage to understand how the ArcMC is doing its calculation

BR,

Hatem

fjdoming Respected Contributor..
Respected Contributor..

Re: Inquiry about ArcMC License Usage data calculation

ArcMC License use for data calculation all inbound traffic in your smartconnectors., If you have winc (connectors) with filter-in activated that traffic isn't considered to calculate the license cosumption, you have observed those differences into distinct values (arcmc vs. loggers).

 

I think you have applied a good filtering out in your smartconnectors, because the difference is greater.

 

 

 

hatemware  Super Contributor..
 Super Contributor..

Re: Inquiry about ArcMC License Usage data calculation

Dear fjdoming and Marijo Mandic

As per the new concept of ADP License pooling:

1- Enforcement no longer done on ADP Logger, so what will happen if the ArcMC records 6 violations in 30 days? any
restrictions apply to the managed Logger or SmartConnectors?

2- the entitled GB/day is across all ADP Loggers as one global entitled GB/day value and accordingly the ingested GB/day
should be calculated across all ADP Loggers as one global ingested GB/day value, so how come the ArcMC is calculating the
ingested rate as the input rate to SmartConnectors? If that is the case, then:
~ how can we utilize the Filteration and Aggregation features to optimize our License usage?
~ do you mean that only WNC connectors and other pull mechanisms' filteration can only be used to reduce ingestion rate
calculation by ArcMC?
~ And any other connector types with pull mechanisms'aggregation (like database) or push mecahnisms's
filtration/aggregation (like Syslog) will not reduce the ingestion rate calculation by ArcMC?
~ If I removed a heavy connector from being managed by ArcMC, does this reduce the ingestion rate calculation by ArcMC? i
think yes but i will lose ArcMC functionality of managing the connector remotely (ex: updates, etc.)

Lastly, as per https://community.softwaregrp.com/t5/ArcSight-Discussions/HOW-TO-ArcMC-manage-a-Logger/td-p/1594233, there
is an option to disable ADP Logger from being managed by ArcMC even if the Logger license type is ADP managed. I think
microfocus introduced that option because of this confusions done to customers, so if i make this i will lose ArcMC
functionality of managing the Logger remotely (ex: updates, etc.) but I will gain the standalone Logger Licensing model
which will calculation consumption after SmartConnector's level filtration/aggregation.

AM I correct? I need your inputs here please on all the above.

BR,

Hatem

fjdoming Respected Contributor..
Respected Contributor..

Re: Inquiry about ArcMC License Usage data calculation

As far as know, currently there aren't any restrictions due to the ArcMC records violations.

I only know  filter-in funtion in WNC connectors,.

Marijo should be able to clear all your doubts, he's the expert.

 

hatemware  Super Contributor..
 Super Contributor..

Re: Inquiry about ArcMC License Usage data calculation

Thanks for your prompt response, 

I hope Marijo can help clear all these doubts 

BR,

Hatem

Highlighted
Micro Focus Expert Martyn Hill
Micro Focus Expert

Re: Inquiry about ArcMC License Usage data calculation

Hi yahnn1 (and Hatem)

I'd like to add the following to what has been said here thus far (you will find many more related questions/answers elsewhere on the forum - esp. under the Customer Support space, under 'Expert Day' - just search there under 'ADP'. You'll need an active Support account associated with your P724 login to gain access to the Customer Support space.)

Firstly, one sould be aware that (today) License Auditing and License Enforcement are handled slightly differently between Software versus Appliance-based Loggers. This difference is liable to change in the future, but for now, regardless of whether ADP or Standalone Logger Licensing is in force, Auditing happens identically between SW and Appliance instances and most signifcantly, event data is never dropped even when running SW/Appliance Logger out of compliance. This is by design and fully aligned with the MF ArcSight paradigm of reliable and full visibility of one's security posture.

Enforcement today is only implemented on the Software Logger and a good description of what features/functions are impacted can be found in the respective Admin Guide. In short, Searching and Reporting are restricted even though event ingesiton continues unaffected.

In the future, we are likely to introduce Enforcement also to the Appliance based Logger as many customers have reported concerns that they might unwittingly go out of compliance with only the Audit mechanism in place to warn them as it is today in Appliance Logger.

In any case, no further impact to ArcMC nor SmartConnector functionality takes place as a result of a license breach.

On to ADP versus 'Standalone Logger' license models:

ADP Licensing measures ingestion differently to how it was under the legacy - or 'Standalone' Logger capacity licensing model. As mentioned earlier in this thread, ingestion under ADP licensing is measured at ingres to the SmartConnectors (and Loggers, for any direct Sylog or similar feeds without using a SmartConnector - not very common.)

This licensing model makes most sense when you consider that ADP now entitles our customers to ingest normalised, enriched and structured event data from SmartConnectors in to both ArcSight and non-ArcSight destinations (or 'consumers', if we use Event Broker terminology.)

Opening our licensing model under ADP to 'anyone else' is one of several key advantages over previous licensing models - more on ADP benefits later.

As has been highlighted, if Filtering and/or Aggregation has been enabled on one or more SmartConnectors - thus reducing the events consumed by their destinations such as Logger, ESM or third-party tools - then you can expect to see a difference between the consumption reports in ArcMC (measuring in ADP terms) versus the Logger license page.

Under ADP, the ArcMC is expected to manage the ingestion capacity entitlement - leveraging the modern 'AutoPass' technology - across the entire ingestion layer, including SmartConnectors, direct Logger feeds and Connectors in Event Broker - CEB.

Thus, when an ADP-entitled license is loaded on to a Logger (post v6.2), the Logger will defer the decision about license 'compliance' to the ArcMC that has been configured to manage its ADP license capacity. Logger will continue to meausure its own ingestion and report it in legacy/standalone terms on its own License page, but this data is no longer used in the compliance decision and can be safely ignored.

In legacy/Standalone Logger licensing on the other hand, there is no requirement to configure the ArcMC to manage the Logger capacity licensing - it remains as before (pre-ADP) and one makes reference instead to the individual Logger license page to determine usage and thus compliance with the purchased entitlement.

All our latest versions of Logger today (and in to the near-future at least) will continue to support both ADP and Standalone licensing models - it is the purchased Entitlement that determines which model applies and upgrades to ADP from Standalone have been made commercially very attractive to assist our customers make the transition.

Now, on to the key differences and benefits between ADP and Standalone Logger capacity Licensing:

Supporting Open Architecture: ADP Licensing is built to support the consumption of enriched event data across the security tooling our customers have deployed - or intend to deploy - ArcSight or otherwise. The pre-ADP licensing model/EULA explicitly restricted consumption of SmartConnector feeds beyond ArcSight products.

Event Broker: With ADP comes the entitlement to our Kafka-based Event Broker (EB) message-bus technology, with its inherent redundancy and scalability capabilities all managed via the enhanced ArcMC web interface. EB is not available under the legacy Logger licensing model.

Ingest Once - Consume Many: Under ADP, it matters not how many times you consume the event-data, only how much was ingested in the first place at the SmartConnector layer. This comes in to play especially where HA Loggers have been deployed - aside from a nominal HA Logger base-license fee, you no longer pay for the capacity consumed in to the HA Logger instances.

Capacity Pooling - WIth ADP, we introduced the modern AutoPass License technology. Whilst 'AutoPass' and 'ADP' are distinct and independent (e.g. it is perfectly possible to run Standalone Logger with an AutoPass license), together they form a key part of our overall solution strategy to ease the management of growing data ingestion at both the commercial and technical layers.

Simplified Licensing metric: By measuring at ingestion, most customers find it much more straightforward to provision licensing for growth - it is almost impossible to pre-deterrmine the final GB/d consumption after SmartConnector Filtering/Aggregation when onboarding new event sources, compared to predicting GB/d produced by any new devices ot be onboarded.

If you're still awake, a final point about recent product features to make ADP adoption even more attrractive.

SmartConenctor Pre-Filtering: In more recent SC releases, we have introduced the pre-filtering capability. ADP licensing is measured after pre-filtering (but before traditional Filtering/Aggregation), augmenting other, device-specific capabilities that can reduce effective ingesiton capacity requirements and thus cost.

Standalone Logger under ADP: For a select few customers, the best of both worlds - all the entitlement of ADP (bar capacity pooling), but side-stepping the need to stand-up an ArcMC instance simply to act as the AutoPass Licensing server for your Logger cluster. This feature was introduced in Logger v6.5 simply to make smaller or remote/restricted site Logger deployments easier by removing the need to have ADP Logger and ArcMC remain in contact. It is especially helfpul to those customers who purchased our latest Gen9 Logger appliance - which was only ever sold under ADP licensing - but never factored-in the need for a separate host to run ArcMC.

I hope that helps :-)

Martyn Hill
Security Customer Success Manager.

hatemware  Super Contributor..
 Super Contributor..

Re: Inquiry about ArcMC License Usage data calculation

Many thanks for your thorough and detailed response

 

Could you please shed more light on SmartConenctor Pre-Filtering because this is the only option that I have to effectively utilize the customer's ADP license and remain compliant.

 

What are the SmartConnector types / mechanism (Push or Pull) that support this feature?

 

You said that this  SmartConenctor Pre-Filtering feature is different than the traditional Filteration and Aggregation features so please advise how to perform it?

 

Another important question, consider this scenario:

 

Multiple Windows Hosts <-- Pull -- Windows Native Connector (WNC) ===>  ESM Express EE7600 & ADP Logger L7600

 

In brief I have only one Windows Native Connector (WNC) that's extracting Security logs from multiple Windows Hosts and then the WNC is dual feeding CEF normalized events to 2 destinations, one is ESM Express and the other is an ADP Logger,
Assume the events per second (EPS) is 2500, and the average raw size is 620 bytes

So what will be the incoming EPS & ingestion rate if I am not using any sort of filtration? Will it be 2500 or the double of 5000 (assuming ESM Express and Logger pull twice)?
and what will be the ingestion rate? Will it be 125 Gb/day = ( [{2500x24x60x60}/{1024x1024x1024}]x620 )?

And what will be the incoming EPS & ingestion rate if I am using filtration of 50% at the ESM Express destination by modifying its runtime parameters via ArcMC and no filteration for the ADP Logger?

And Finally what will be the incoming EPS & ingestion rate if I am using filtration of 50% at both destinations by modifying their runtime parameters via ArcMC?

I really need answers on the above as I am struggling since long time to understand this new ADP licensing model.

BR,
Hatem

Micro Focus Expert Martyn Hill
Micro Focus Expert

Re: Inquiry about ArcMC License Usage data calculation

Hi Hatem!

Good questions. Let's see what I can address myself:

Q: What are the SmartConnector types / mechanism (Push or Pull) that support this feature?
A: As this is a framework feature, ALL types of SmartConnector can levarage pre-filtering (aka 'Customised Event Filtering')

 Q: You said that this  SmartConnector Pre-Filtering feature is different than the traditional Filteration and Aggregation features so please advise how to perform it?
A: Your best bet is to review the latest 'SmartConnector User Guide' - page 54 under the heading "Customized Events Filtering". In short, as the filtering is applied directly on the original Raw Event, you define Regex include/exclude clauses to filter-in OR out respectively. You add two or three new properties directly in the agent.properties file.

Q: Assume the events per second (EPS) is 2500, and the average raw size is 620 bytes. So what will be the incoming EPS & ingestion rate if I am not using any sort of filtration? Will it be 2500 or the double of 5000 (assuming ESM Express and Logger pull twice)?
A: As ADP ingestion measures ingestion only once - not by how many times it is consumed - with no Filtering/Aggregation enabled, the relevant ingestion figure is 2,500 EPS.


Q: ...and what will be the ingestion rate? Will it be 125 Gb/day = ( [{2500x24x60x60}/{1024x1024x1024}]x620 )?
A: Correct, The ADP license woulkd need to meet or exceed 124.723 GB/d

Q:And what will be the incoming EPS & ingestion rate if I am using filtration of 50% at the ESM Express destination by modifying its runtime parameters via ArcMC and no filteration for the ADP Logger?
A: ESM Express would need to be licensed for 2500 * 50% = 1,250 EPS. ADP remains at 125 GB/d regardless - what you choose to send to Logger is no longer relevant and the same ADP ingestion would entitle you to send to 2, 3 or 30 Loggers - with only a base-instance License required for each ADP Logger instance you deploy - you've already paid for the ingestion capacity!

Q: And Finally what will be the incoming EPS & ingestion rate if I am using filtration of 50% at both destinations by modifying their runtime parameters via ArcMC?
A: Same as the above answer... Much simpler than previous licensing models, right? Given that filteration/aggreagtion needs change based on the ever evolving use-case requirements (and the required source devices), removing this unknown from your provisioning plans is a breath of fresh air...

The way to understand ADP licensing is to make the mental 'switch' away from considering Logger consumption and instead to ingestion in to the platform as a whole. In ADP, Logger is simply an advanced set of features and functions focussed on Compliance, data-immutability and long-term retention and Reporting that is built-in to ADP - alongside everything that ArcMC has to offer.

M.

hatemware  Super Contributor..
 Super Contributor..

Re: Inquiry about ArcMC License Usage data calculation

Many thanks Martyn for your detailed explaination, you are really a good example for Microfocus.

I will try to use this customized prefiltering feature with exclude property
BR, Hatem