Streaming performance data with Operations Bridge - II

[GaneshAgrahara]

Co-authored by Roy Kallol, ITOM Engineering team

By now you should be aware of the performance management capability of the Operations Bridge suite. If you missed the first part of this series, which explains how you can stream metrics from a single node using Operations Agent policies, you can read it here.

This blog post provides information about how you stream metrics from many nodes - that have been collected by a third-party system - using the Operations Connector.

For this you can use any one of the following policy types for the integration of metrics / performance counters:

Database Policies:

Database policies enable you to collect data from database tables used by third-party systems by performing a query through a JDBC connection.

Perl Script Policies:

Perl script policies enable you to collect data from systems by periodically running Perl scripts which perform the actual data collection.

Structured Log File Policies:

Structure Log File policies read entries in a log file and matches the desired text against the log file structure fields defined by using the Operations Manager (OM) pattern-matching language. When the policy finds a match and certain conditions apply, the policy responds by sending metrics data.

REST Web Service Listener Policies:

REST Web service policies receive and process data from third-party systems through the Operations Connector REST Web service listener since collection is done outside.

XML File Policies

XML file policies read values in XML files and respond when the value that you choose appears in the file. XML file policies are especially suited for integrating data from third-party systems that can store their information in XML format.

Micro Focus offers several Operations Connectors that provide metric collection out of the box, so that you don’t have to write the collection policy yourself.

The table below shows some of the out-of-the-box connectors along with the metric policy type they use.

Note: The Metric Streaming Configuration policy also allows you to collect metrics but it is strictly not advisable for third-party connector use cases or any other large scale integration solutions where the number of data points being collected/received is on a large scale. The Metric streaming configuration policy should be used only for integrating metrics from a single node.

Once metrics are collected using one of the above policy types by the Operations Collector, they can be streamed to the Operations Bridge Performance Engine using a Data Forwarding policy.

Example: Operations Connector for SCOM metric streaming

Let’s do a deep dive for Operations Connector for SCOM. The below diagram shows the overall solution flow of third-party metric data streaming by configuring the end target as Performance Engine.

The Data Forwarding policy is defined in Operations Manager i and deployed to the managed connector nodes. The Operations Agent streams the metric data to the OMi Performance Engine which acts as the central place for storing and analyzing metrics in a large-scale environment. OMi Performance Dashboard reads the metric data from the OMi Performance Engine and displays the metrics on the performance dashboard.

Why is this not in real-time for third-party connectors?

Data streamed from third-party connectors is never in real time. (Figures will depend on specific implementation characteristics. Please refer to the Performance and Sizing Guide for details.) The lack of real-time results is due to the fact that the data is from a third-party connector and the Operations connector doesn’t have control on the collection performed by the domain manager (even if collection is done in pull or push mode). There is always a gap between the time the data was collected and the time the data is displayed on the Performance Dashboard.

Configuration Steps

1. Configure the SCOM Domain Manager aka SCOM server
2. Install BSM/Ops Connector and Ops Connector for Microsoft SCOM
   For SCOM REST Web service listener policies process the data received by the Ops Connector REST Web service listener.
3. Make sure that Operations Agent version 12.03 or later is installed
   Note: Operations Connector by default comes with Operations Agent installed.
4. Disable the local metrics store

Since you are storing the metric data in the central store from which the Performance Dashboard gets the data for graphing, you don’t have to duplicate it by storing it on the Ops Connector node.

By disabling data logging to the local store, you can avoid performance degradation of Operations Agent’s collector component “OACORE”.

It’s advisable NOT to log more than 100 data points per second into the local metric store.
As the Operations Connector for SCOM integrates metrics data for many nodes, the local metric store should be disabled.

(Number of data points = Number of Managed Nodes x Number of Classes x Number of Metrics x Number of Instances)

Execute the following command to set OPCGENI_DISABLE_LOCAL_STORE to “true”
ovconfchg -ns eaagt -set OPCGENI_DISABLE_LOCAL_STORE true

Once the variable is set, restart the Operations Agent.
After setting the parameter to true, the local data logging of METRICS into the SQLite DB of Operations Agent is disabled. This affects all METRIC integration policies using the policy types Database, Perl, Structured Log file, Rest Web Service Listener or XML file.
Other policy types (for metric, event or topology integration) are not affected by this setting.

1. Edit and Configure the Data Forwarding policy

With Data Forwarding policy, the data points are being streamed by OPCGENI. Whereas with Metric streaming configuration policy, the data is streamed by HPSENSOR.

Configure Targets

You can also define the rules on what data is sent to which target or to discard specific data for some targets. The example shown below forwards metric data corresponding to data Domain named BsmIntSCOMMetrics.

 Configure Rules

6. Configure the Performance Engine Node details from which Performance Dashboard must request data.

 7. Create a SCOM Performance Dashboard in OMi

See Create a Performance Dashboard and Configure Performance Dashboard

The SCOM connector also integrates topology into OMi. Select a CI created by the SCOM topology integration (for example a node CI), then select the Data source OpsBridge Store. You should then be able to select a Class Name, followed by Metric Name and Instance Name of metrics that have been forwarded to the OpsBridge Store aka the Operations Bridge Performance Engine.

 Note: Depending on the type of Operations Connector (out of the box), install the OMi Content Pack (which includes Topology Synchronization files, performance dashboard and other configuration) for the respective Ops Connector (if available) followed by enabling the respective Topology Integration policies. Once done, select a view containing the CI for creating the dashboard.

 Graph plotted from historical data

You can save the respective performance dashboard as a ready-to-use template for the future.

Large scale metric streaming using multiple collectors

Using the same concept and configuration steps as above, performance data can be streamed also from multiple third-party systems, such as SCOM and Tivoli.

Note: The below are the minimum Operations Bridge capability versions required to use the metric streaming functionality to stream third-party data.

• Operations Manager i (includes OMi Performance Dashboard) 11
• OMi Performance Engine 11
• Operations Agent 03
• Respective Operations Connector

Events

To get more information on this release and how customers are using Operations Bridge we are happy to announce the following events you can register for

Vivit event – Operations Special Interest Group Webinar – 11^th October 2017 – Register here
ITOM Customer Forum, Copenhagen – October 3^rd 2017 - Register here
ITOM Customer Forum, Stockholm – October 4^th 2017 - Register here
ITOM Customer Forum, London – October 5^th 2017 - Register here
ITOM Customer Forum, Brussels – December 7^th 2017 -  coming soon

Read all our news at the OpsBridge blog

References

Explore all the capabilities of the Operations Bridge Suite and technology integrations by visiting these sites:

• Operations Bridge Suite
• Operations Manager i
• Operations Bridge Technology Integrations eBook
• Operations Bridge Integrations page

[mdcomputerguy]

If you are sending SCOM or Zabbix performance data to the engine how long is data retention with and without Vertica?

[Carol Park]

OMi Performance Engine (PE) stores 720 data points.  Thus if you send data to it every 5 seconds, that equates to 1 hour of data.  If you send data to it every 10 seconds, that equates to 10 hours of data.  And so on....  eg 5min data would mean PE retains 2.5 days of history.

If you then integrate PE with Vertica, then Vertica stores 30 days of data by default.  This value is configurable.

CP.