Application Perf Mgmt (BAC / BSM) User Discussions
cancel

Sitescope using up CPU

Soderborg Respected Contributor.
Respected Contributor.

Sitescope using up CPU

Hi,

I have a problem with Sitescope using up 100% CPU and have >200.000 handles after a couple of days uptime.
When this occur, Sitescope is not able to provide any metrics.

Monitor runs per minute: ~1200
Unix servers monitored: ~100
Windows servers monitored: ~10
Total license points 500; points consumed 480

Sitescope Configuraiton Tool Sizing changed the default config to JVM size to 4096MB, desktop heap size to 8192MB and number of handles to 18,000.

Installation spec:
Windows 2008 sp1 64bit running as a VMware VM (12 GB RAM and 4 cores at 2.30GHz Intel Xeon E5-2670v3)
HP SiteScope 11.33.381 64-bit JVM, Build 290

We have ~10 JMX monitors that often fails (my main suspect for this issue)
There are also several scripts that collects metrics from different Java processeses.

Anyone here with the same issue and perhaps some guidelines on how to solve it?

Thanks

7 REPLIES
dasomm  Super Contributor..
 Super Contributor..

Re: Sitescope using up CPU

Hi,

Have you enabled "Verify on Error" on some/many monitors?
How are the numbers for SSH failed to open V1/V2? 
Are your monitors running with very high frequency?
Any instances of 'monitors waiting'?

 

 

 

 Honored Contributor... Tarif Louah  Honored Contributor...
 Honored Contributor...

Re: Sitescope using up CPU

Make sure this not an issue introduced by the OS (or missing an OS patch).

Aslo, Pls refer to key point in Using SiteScope 11.33:

One key warning signal for interpreting monitoring load is the ratio of Monitors Waiting to Monitors Running.
Generally, having some monitors waiting for execution is not a problem unless the ratio of Monitors Waiting to
Monitors Running is consistently 1:2 or higher. For example, if the number of monitors running is at the
maximum of 100 and there are 50 monitors waiting, this represents a ratio of 1 monitor waiting for every two
running.

The note about "initial maximum number of monitor execution threads" setup is also good.

Soderborg Respected Contributor.
Respected Contributor.

Re: Sitescope using up CPU

Hi,

Thanks for your answer.
Here's the feedback to the questions you asked.
Do you have any suggestions on how to proceed or scale our Sitescope?

Thanks

Have you enabled "Verify on Error" on some/many monitors?
- Nope, not on any of the monitors

How are the numbers for SSH failed to open V1/V2?
- Total opened    196
- Total closed    110
- Total cleaned    62
- Total failed to open V1    0 (Only using SSH v2)
- Total failed to open V2    5 806
- Reused    10 979 136
- Currently allocated resources    86
- Currently in use    5
- Average call time for last 10 minutes    22
- Total average call time    22

Are your monitors running with very high frequency?
- Yes, I guess we do. Frequenzy is set to 10-15s.
We want statistics very often when running performance tests. When Garbage Collections occur, we can correlate that to response times, when CPU suddenly hits 100%, we can correlate that to what application feature is being invoked etc.

Any instances of 'monitors waiting?
- Monitors running    Current: 4    Max: 82
- Monitors waiting    Current: 0    Max: 20
- Monitors runs per minute    Current: 1 196,1    Max: 1 219,9

Soderborg Respected Contributor.
Respected Contributor.

Re: Sitescope using up CPU

Hi Tarif,

Thanks for your answer.

The OS (Win2008 R2 Standard 64bit) is up to date with patches.

Monitors waiting vs. Monitors running semes to be OK. Max 82 running and max 20 waiting. Normally 2-5 running and 0 waiting.

Sitescope Server settings:
Maximum monitor processes: 100
Maximum monitor running: 400
Minimal monitor run interval: 15 s

So it looks to me that we do not have any issues there. Do you have any other opinion or suggestion?

Thanks
/Fredrik

 

 Honored Contributor... Tarif Louah  Honored Contributor...
 Honored Contributor...

Re: Sitescope using up CPU

Ok fewer options here really, assuming Sis heath was checked and skipped monitors are reasonable. You may want to identify which monitor/group causing these spikes:

-stop SiteScope, disable all monitors (there is a flag in mster.config), start sitescope and enable monitors by group .. 

- use Jconsole.exe to detrmine which threads consume high CPU ..

- aletrnatively, export configuration reinstall Sitescope and import config again or use different machine (trial&error).

other than that open support case to be investigated.

Thanks,

Tarif 

 

Highlighted
dasomm  Super Contributor..
 Super Contributor..

Re: Sitescope using up CPU

I would focus on this one first and try to find the cause:

- Total failed to open V2    5 806

Generally it's good to have this value as low as possible, and with the frequency you run I would think SiteScope is especially vulnerable to increased wait time and retries regarding logon to servers.

 

ADPRJ  Valued Contributor..
 Valued Contributor..

Re: Sitescope using up CPU

Another option to verify:

For your JMX monitors. I have quite a bit more than you, which I had to define a domain filter to reduce the # of counters selected and the size of the xml tmp files created by these monitors in SiteScope/templates.applications folder. Check the file sizes. This really slowed down my scope. 

Top Contributors Last 30 Days