Data Protector User Discussions
cancel

NDMP Variable

Highlighted
Silva Trusted Contributor.
Trusted Contributor.

NDMP Variable

Hi,

I found this variable related to NDMP performance: OB2NDMPCATQUESIZE., with the definition below:

variable sets the number of internal buffers that hold catalog information before storing it to file history swap files. By fine tuning the value, you can increase, to a certain extent, NDMP backup performance.
When set to 5, the NDMP Media Agent can process up to 20 million files (in one backup specification), provided that enough system resources are available (approximately 1.9 GB of system memory and 2.8 GB of disk space).
Set the variable to higher values if the number of files in the backup specification is less than 20 millions and enough system memory is available.

 

Actually I got confusing when it said that: "Set the variable to higher values if the number of files in the backup specification is less than 20 millions" --> So does that mean we can increase the buffer IF ONLY the number of file less than 20 million? What is the maximum number of this value?

If we have a very large backup spec, let say 100 million files, how much value of this variable we should set? (assume the system have enough memory to handle)

How this variable effect to the performance of backup?

Any idea would be appreciated,

5 REPLIES
Micro Focus Expert
Micro Focus Expert

Re: NDMP Variable

Hello @Silva

According omnirc info these are the details of the variables: 

# OB2NDMPCATQUESIZE=n
# Default: 5
# NDMP MA-specific environment variable.
# This variable is used to fine-tune memory and disk consumption and overall
# performance of the NDMP backup. When the variable is set to default, the NDMP MA
# can process up to 20 million files (in one backup specification) if sufficient
# system resources are available (approximately 1.9 GB of system memory and 2.8 GB of
# disk space). If sufficient system memory is available and the number of files in a
# backup specification is less than 20 million, the variable can be set to a higher
# value. Increasing the value of the variable will increase the performance of
# NDMP backups to a certain extent. The value of the variable represents the number of
# internal buffers that hold catalog information prior to writing it to file history
# swap files. Memory allocation overhead in kilobytes can be calculated by
# multiplying the value of the variable by 512.

So, if you have more files than 20 million you should set the variable with less value. 

Regards, 

Andres Fallas Salazar
Customer Support Engineer

If you find that this or any other post resolves your issue, please be sure to mark it as an accepted solution.
If you are satisfied with anyone’s response please remember to give them a KUDOS by clicking on the STAR at the bottom left of the post and show your appreciation.
Silva Trusted Contributor.
Trusted Contributor.

Re: NDMP Variable

Hi Victor,

Are you saying that if I have over 20 million files then the maximum value can set is 5?

Micro Focus Expert
Micro Focus Expert

Re: NDMP Variable

Hello @Silva

Actually you can increase if you have less than 20 million. If you have more than this, you can decrease the value maybe to 3 or 2. 

Then check the behavior, 

Andres Fallas Salazar
Customer Support Engineer

If you find that this or any other post resolves your issue, please be sure to mark it as an accepted solution.
If you are satisfied with anyone’s response please remember to give them a KUDOS by clicking on the STAR at the bottom left of the post and show your appreciation.
Silva Trusted Contributor.
Trusted Contributor.

Re: NDMP Variable

Thank Victor,

Actually I not got the logic of variable instruction, when I decrease the variable then the memory of buffer will decrease right? so what is the value used for? while if we have more than 20 million files we already may have slower performance,

And another thing, may I know which else NDMP variable need to adjust to deal with a backup spec which contains over 100 million files? (to avoid the issue "NDMP DA run out of memory")

Micro Focus Expert
Micro Focus Expert

Re: NDMP Variable

Hello @Silva

Try to decrease the value of the variable for bigger environments. Variable OB2NDMPCATQUESIZE should be set in less value if you have a lot of files. 

Thanks for the error, this clarify me more the situation. For this error try to add these variables on NDMP MA: 

  • OB2NDMPMEMONLY=0

# If the variable is set to 0, the NDMP MA will store part of the catalog
# information in file history swap files. For more information on file history
# swap files see the OB2NDMPFHFILEOPT variable description. The OB2NDMPMEMONLY
# variable must be set to 0 if the number of files in a backup specification
# exceeds 5 million. Setting the variable to 0 will allow the NDMP MA to handle
# backups of up to 20 million of files (in one backup specification) if the
# system has enough resources; to back up of 20 million files, approximately
# 1.9 GB of system memory and 2.8 GB of disk space is needed.

  • OB2NDMPFHFILEOPT=<path>, <max_number>, <max_size>

# For example:
# on Windows: C:\tmp, 32, 1024; D:\tmp\tmp_1, 10, 1024
# on UNIX: /tmp, 10, 1024; /var/tmp, 5, 60
# The file history swap files can grow very large. The formula below can be
# used to calculate approximate disk consumption by file history swap files:
# EstConsumption = (NumofFiles+NumofDirs)*(136+AverageFileNameSize)
# Where NumofFiles is the number of backed up files and NumofDirs is the
# number of backed up directories.

I think this could help you more. 

Regards, 

Andres Fallas Salazar
Customer Support Engineer

If you find that this or any other post resolves your issue, please be sure to mark it as an accepted solution.
If you are satisfied with anyone’s response please remember to give them a KUDOS by clicking on the STAR at the bottom left of the post and show your appreciation.