NOTICE: Branded Content
NOTICE: Certain versions of content (“Material”) accessible here may contain branding from Hewlett-Packard Company (now HP Inc.) and Hewlett Packard Enterprise Company. As of September 1, 2017, the Material is now offered by Micro Focus, a separately owned and operated company. Any reference to the HP and Hewlett Packard Enterprise/HPE marks is historical in nature, and the HP and Hewlett Packard Enterprise/HPE marks are the property of their respective owners.
Data Protector Idea Exchange

Backup a single file system with multiple streams

Brief description:

While file systems continue to grow and the underlaying disk subsystems become faster the backups of large file systems are still one of the major concerns for traditional backup environments. The Data Protector Disk Agent should support multiple streams on a single file system that can be configured similar to device concurrency to drastically speed up the backup and the restore operation.


Existing Enhancement Requests:

QCCR2A62413: Support multiple disk agents per volume


Reduce the complexitiy of subdividing file systems manually into smaller pieces. Optimize for backup and restore performance using multiple streams even on large file systems with millions of files and folders.

 Super Contributor..

With today's multi-TB filesystems, this feature is needed more than ever.  I wrote the original "divide-and-conquer" article back in 2011:

After the first tree walk, a disk agent (VBDA) should have all the info needed to intelligently analyze the filesystem and execute a dynamic divide-and-conquer strategy to read files/folders in an optimal, parallel fashion.

The potential downside is that such VBDA stream multiplexing would adversely affect inline dedupe which relies upon a single, non-multiplexed data source per media agent (BMA) stream headed to a Catalyst store.  I suppose assigning a unique BMA per VBDA "substream" would be a viable workaround.

Another consideration is not going crazy with the # of VBDA threads/streams.  The filesystem being read by VBDA lives on a LUN that can easily be pushed to its limit of concurrent IO.  You'll likely reach a LUN's concurrent IO limit with a half dozen streams or less.  In my experience using manual divide-and-conquer, here's what I've found with parallel streams from one filesystem:

2 streams better than 1?  Always.
3 streams better than 2?  Quite often.
4 streams better than 3?  Usually but not always.
5 streams better than 4?  Less often - depends on the underlying disk storage.
6 streams better than 5?  Almost never because you've saturated the concurrent IO capability fo the underlying LUN.

The point of diminishing returns is generally five or six parallel streams coming from the same filesystem - again depending upon a plethora of factors spanning storage, infrastructure, and host resource capability.

Outstanding Contributor.
Greedy approach to division is SOP for integration agents that have this many-to-many mapping from files (e.g. Lotus), so this suggestion is not without precedent. @Jim Turner: Per B2D design in DP, each stream goes to own medium which translates to one object-store item, and deduplication is scoped to entire store, not single item. So, DP already assigns unique BMA per stream, and the only thing needed is for VBDA to have multiple streams.
 Super Contributor..

Thank you, @antaln.  Yes, per B2D design in DP, each VBDA has a 1:1 relationship with a BMA, effectively disabling source multiplexing.  My worry is that allowing a VBDA to multi-thread and send parallel streams of data to its assigned BMA would go against that design spec.  The success of HPE's proprietary TTTD algorithm used by the Catalyst client within the BMA to chunk data before it's hashed relies heavily upon the source data being a single, non-multiplexed stream enabling the detection and elimination of duplicate 4 KB (on average) chunks.  If a single VBDA is feeding a single BMA multiple streams of data in parallel, the unpredictable mix of data from each backup session would heavily degrade dedupe performance.  It is not a function of whether or not the Catalyst store reperesents the dedupe domain (hash database).

This situation can be mitigated by ensuring that each VBDA thread is assigned its own BMA in the future, should the DP VBDA gain enough intelligence to analyze and dynamically subdivide a large filesystem into parallel parts to read simultaneously.  It would also be my suggestion that any such feature also limit the number of parallel VBDA "substreams" if you will to no more than 4 or 5.  I say that because I doubt seriously we would add enough intelligence to VBDA to enable it to monitor the SCSI queue depth and utilization at the OS level to know when excessive streams are putting the host's IO subsystem in distress.

Outstanding Contributor.
@Jim Turner: agreed on all points. The design of the data format in DP prohibits one source (=one daid) to multiplex data - the data records are expected to occur in series and to be contiguous. @Sebatian.Koehler: what exactly did you mean by "threads per stream" setting in your proposal?
Knowledge Partner

Hi antaln,

If we focus on the Disk Agent for a moment I see two possible ways of optimizing throughput.

Multiple Streams:
Automatic subdividing of the source volume into multiple steams (e.g. based on directory structures) which can be concurrently processed. Similar to what we can configure today manually, but the data streams must be combined into a single entity in the IDB to maintain simple restore (e.g. drive D: was backed up with 4 streams, the IDB should only list one object for drive D: / today we will have 4 entities with a subset of the data each). Each data stream can be send to a BMA (with concurrency of 1 or more) depending on the device type.

Multiple Threads:
Each data stream could benefit from being multithreaded. Meaning multiple threads are fetching data from the disk in parallel. Robocopy for example has a similar option to speed up the processing rate.

In general I would leave the details to R&D so they can do the math, benchmarking and consider possible effects on backup devices such as StoreOnce.

Sebastian Koehler

 Super Contributor..

Don't rely on folder structure for making the automatic splits.  I've seen some filesystems with no subfolders that contain thousands of files amounting to hundreds of GB.  It's usually some wonky imaging app that stupidly dumps all output in to one folder with no option of dividing into subfolders by job, date, etc.

Knowledge Partner

Hi Jim,

This is true. I know how challenging it can be when manually configuring multiple streams. File Systems can have all kind of structures. With no folders or tousands of of folders in the root. While multi-streaming of a block based backup is realitivly simple to implement (get the size, process pieces of similar sizes in parallel) we're looking for ways to optimize a traditional file system based backup. This means we have to work with the directory structures the Disk Agent can find.

I would try to get the overall volume usage first, fetch top level folders and their usage. From there decide if the 2nd/3rd level folders need to to be considered or not. Each folder is processed as a stream where we can read data using a single thread or multiple threads.

Sebastian Koehler

Outstanding Contributor.

Greedy distribution is simplest and self-balancing: assign next file to stream that is available for processing. This precludes any thinking about what the "ideal" distribution would be, but the cost of possibly splitting files from same directory into multiple streams, thus potentially prolonging the restore process.


One problem with all multi-stream approaches in DP is that user is forced to pick the number of streams. But that number cannot be correct for all situations: e.g. using 5 streams for incremental backup that picks up only 5 small files is wasteful. And the number of streams can conflict with number of destination devices that are selected or available, leading to situations where backup cannot continue (you selected 5 streams, but device can only handle 3). So maybe the setting should specify an upper limit to prevent diminishing returns described by @Jim Turner, but the actual number up to that limit is then inferred from the selected destination devices and their capacity for concurrency at the time of backup?


@Sebastian.Koehler: How would you expect related functionality to behave in presence of multiple streams: object consolidation, object copy, protection?

Micro Focus Contributor
Status changed to: Under Consideration

This idea is being reviewed by engineering to determine the feasibility of this request. Will revert back shortly.

 Micro Focus Expert

@Sebastian.Koehler- I have the same question as @antaln. What would be your expectation around the split streams with regards to features like object copy and object consolidation? Should the pieces have an indepedent existence or should they all be tied together under the parent entity?