SOS object copy will start just one media agent

Hi,

I have configured an object-copy-job between to StoreOnce Software Stores. Each store has two gateways (max. parallel streams per GW: 8).
When the copy-job is running, only ONE media agent is started per gateway. Running a backup job with one of these stores as a target will start media agents until the configured maximum has been reached.
omnirc has no media agent specific settings on the store-/gateway-servers.
DP version: 9.06

I'm wondering, what's wrong with the copy job.

In another DP cell (similar setup) during the copy job media agents are started until the configured maximum has been reached (works as expected). DP version 9.06-9.08
Comparing the store- and gateway-configurations shows no differences.


regards
Joerg

Parents
  • Hello Joerg,

    Looking at your query, which I can see to be specific to concurrency/performance, I can confirm that "With StoreOnce software
    deduplication, Disk Agent concurrency for each Media Agent is set to 1".

    I will need to refer you to the documentation for complete technical discussion.

     =================================================================================

    https://softwaresupport.hpe.com/km/KM01024691/hpe_wp_DP9.07_Deduplication_pdf.pdf

    Technical white paper
    HPE Data Protector 9.07 Deduplication
    Introducing Backup to Disk devices and deduplication

    Target-side deduplication

    The deduplication process takes place on the target device (3). It receives data to be backed from Media Agents
    installed on clients (gateways).

    Target-side deduplication using the StoreOnce Software system

    The StoreOnce Software deduplication system then writes the deduplicated data to the StoreOnce library (this is the
    physical store and is sometimes referred to as the deduplication store).

    The StoreOnce software deduplication system allows connections from several Media Agents, locally or remotely. It
    also provides synchronization mechanisms to enable multiple Media Agents to work with the StoreOnce library at
    the same time. The Media Agent reads or writes data in terms of object versions to or from the StoreOnce library.
    Each object version is represented as an item in the StoreOnce library. To optimize deduplication performance, Disk
    Agent concurrency is not supported (this means, one Disk Agent talks to one Media Agent – there is no multiplexing
    of streams). An example configuration showing a basic local and remote office deployment is given in Example
    configuration using a B2D device.

     

     Concurrency specifies a number of Disk Agents writing to the device in parallel. Multiple Disk Agents read data in
    parallel (from disks) to provide a constant data stream to the Media Agent. With StoreOnce software
    deduplication, Disk Agent concurrency for each Media Agent is set to 1 (this improves the deduplication ratio).

     =================================================================================

     

    Hope this helps

    Rgds,

    Sherman

  • What I'm seeing is that during backup multiple MA's are getting started in parallel, but in your copy this is not happening. So, it's not related to gateway configuration, but rather to the way objects are being selected and started during the copy. Maybe you should start with sharing a session report and copy spec of the copy session. The better option is to generate debugs and provide those to support.

    Koen

  • Joerg,

    Ales was giving a key hint! But you didn't really answer his question. What was the load balancing set to during the backup session?

    I'm reproducing the behavior by running a backup with "1 1" load balancing _AND_ "Single Object per Store Media" on the device disabled. This is resulting in a backup using just 1 gateway, backing up multiple objects serially to the same slot/medium. Duirng the copy this results in 1 reader and 1 writer copying the objects and this with load balancing "1 5" set. The question now is: how were your objects written duirng the backup and to which slots?

    I hope this helps you to come to a conclusion.

    Koen

  • Hi,

    As Koen already suggested. In order to understand why copy is using only certain number of MAs, we need to know how backup was performed. Backup can influence restore/copy in multiple ways.

    So please provide backup “datalist”, “session report” and both devices being used (backup and copy) details.

    Regards AlesKol

  • Hi Koen, hi AlesKol,

    sorry, I really missed that part of AlesKol's question.
    I see what you mean and I double checked the backup job. Here the BACKUP-job infos:

    Devices used for backup/object copy:
    ==========================
    Both SOS-Stores are configured with 2 Gateways and "Single Object per Store Media" enabled.
    Max. Connections per Store: not set
    Max. Connections per GW: 8

    Backup datalist:
    ===========
    BARLIST "sbztnv0818_Infrastruktur_B2D"
    GROUP "VMWare"
    DYNAMIC 1 6
    DEVICE "B2D_BZT_VM_gw-dpma03"
    {
    -sync
    }

    DEVICE "B2D_BZT_VM_gw-dpma04"
    {
    -sync
    }

    CLIENT "/BZT-B" dpma04.hwk.local
    {
    -exec vepa_bar.exe
    -args {
    "backup"
    "--virtual-environment"
    "VMWare"
    "--readstdin"
    }
    -input {
    "method=4;preexec='';postexec='';origviewmode=7;appserver='sbztnv0818.hwk.local';mounthost='';free_space_required=10;include=({type=4;path='/BZT-B/Cluster_BZT/Exchange_Data';uuid='5021fcac-9449-95fc-8611-5bbcbdaefa61';disks=({name='scsi0:0';});},{type=4;path='/BZT-B/Cluster_BZT/Exchange_Acc';uuid='5021efac-9e68-634e-376e-8c4125877575';},{type=4;path='/BZT-B/Cluster_BZT/AV_ID_ENGINE_02';uuid='5018997a-3ce8-54e5-ff80-49e5d5a4f8cd';},{type=4;path='/BZT-B/Cluster_BZT/Facility-Mgm';uuid='50187015-41ac-70dc-f6a6-19492f30e146';},{type=4;path='/BZT-B/Cluster_BZT/ZEUS';uuid='52029190-1858-468a-684d-dbc767f0f630';},{type=4;path='/BZT-B/Cluster_BZT/Dpilinux';uuid='50214542-7833-e005-1cee-52e1c8f1d3fa';},{type=4;path='/BZT-B/Cluster_BZT/theGuard_SMC';uuid='50213439-1ce7-76d1-ac44-3ff8adf7071c';},{type=4;path='/BZT-B/Cluster_BZT/ID Audit Tracker';uuid='50294d94-fc26-bb9d-2eab-bd075c348996';},{type=4;path='/BTT-B/Cluster_BZT/Avaya-WOS';uuid='50187229-705f-16dc-0c4c-0fee3202670c';},{type=4;path='/BZT-B/Cluster_BZT/8MAN';uuid='501889e6-c7d8-9700-f8c6-787578020efc';},{type=4;path='/BZT-B/Cluster_BZT/Management';uuid='52fc82f0-b287-8ff0-1a3d-89427e7a49da';},{type=4;path='/BZT-B/Cluster_BZT/mysql_clone';uuid='5021b1b1-8b97-4866-98a0-5b484465a3fc';},{type=4;path='/BZT-B/Cluster_BZT/SBZTNV0868.cust.local';uuid='5018c422-4f62-c950-42ce-ba0fe722b182';});"
    }
    -public
    } -protect weeks 2

    Backup session report:
    ================

    [Normal] From: BSM@dpcm01.cust.local "sbztnv0818_Infrastruktur_B2D" Time: 27.11.2016 16:30:10
    Backup session 2016/11/27-8 started.

    [Normal] From: BSM@dpcm01.cust.local "sbztnv0818_Infrastruktur_B2D" Time: 27.11.2016 16:30:10
    OB2BAR application on "dpma04.cust.local" successfully started.

    [Normal] From: VEPALIB_VMWARE@dpma04.cust.local "/bzt-B" Time: 27.11.2016 16:30:11
    Resolving objects for backup on vCenter 'sbztnv0818.cust.local' ...

    [Normal] From: VEPALIB_VMWARE@dpma04.cust.local "/bzt-B" Time: 27.11.2016 16:30:15
    Add Virtual Machine to the backup ...
    Name: Exchange_Data
    Path: /bzt-B/vm/Kommunikationssysteme/Exchange_Data
    InstanceUUID: 5021fcac-9449-95fc-8611-5bbcbdaefa61

    [Normal] From: VEPALIB_VMWARE@dpma04.cust.local "/bzt-B" Time: 27.11.2016 16:30:17
    Add Virtual Machine to the backup ...
    Name: Dpilinux
    Path: /bzt-B/vm/Dpilinux
    InstanceUUID: 50214542-7833-e005-1cee-52e1c8f1d3fa

    ... (removed some lines with "Add virtual machine..."/"Locking vMotion...")

    [Normal] From: VEPALIB_VMWARE@dpma04.cust.local "/bzt-B" Time: 27.11.2016 16:30:53
    Virtual Machine 'Dpilinux': Locking vMotion ...

    [Normal] From: VEPALIB_VMWARE@dpma04.cust.local "/bzt-B" Time: 27.11.2016 16:30:55
    Creating folder C:\ProgramData\OmniBack\tmp\c3ef987b-1134-4340-a299-3955b95486a3 ...

    [Normal] From: VEPALIB_VMWARE@dpma04.cust.local "/bzt-B" Time: 27.11.2016 16:30:55
    Creating folder C:\ProgramData\OmniBack\tmp\ef01bb77-5eff-4038-8aad-cb1603dfcb47 ...

    ... (skipped some lines)

    [Normal] From: VEPALIB_VMWARE@dpma04.cust.local "/bzt-B" Time: 27.11.2016 16:32:04
    Virtual Machine 'Guard_SMC' (UUID '50213439-1ce7-76d1-ac44-3ff8adf7071c', Disk 'scsi0:0'): Starting full backup ...

    [Normal] From: BMA@dpma04.cust.local "B2D_bzt_VM_gw-dpma04 [GW 6384:0:3941272586470069107]" Time: 27.11.2016 16:32:05
    STARTING Media Agent "B2D_bzt_VM_gw-dpma04 [GW 6384:0:3941272586470069107]"

    [Normal] From: BMA@dpma04.cust.local "B2D_bzt_VM_gw-dpma04 [GW 6384:1:3941272586470069107]" Time: 27.11.2016 16:32:05
    STARTING Media Agent "B2D_bzt_VM_gw-dpma04 [GW 6384:1:3941272586470069107]"

    [Normal] From: BMA@dpma04.cust.local "B2D_bzt_VM_gw-dpma04 [GW 6384:0:3941272586470069107]" Time: 27.11.2016 16:32:06
    Loading medium from slot \\dpma04.cust.local\B2D_bzt_VM\c80010ac_583afc76_0858_39e1 to device B2D_bzt_VM_gw-dpma04 [GW 6384:0:3941272586470069107]

    [Normal] From: BMA@dpma04.cust.local "B2D_bzt_VM_gw-dpma04 [GW 6384:1:3941272586470069107]" Time: 27.11.2016 16:32:06
    Loading medium from slot \\dpma04.cust.local\B2D_bzt_VM\c80010ac_583afc76_0858_39e2 to device B2D_bzt_VM_gw-dpma04 [GW 6384:1:3941272586470069107]

    [Normal] From: OB2BAR_VEPA_BAR@dpma04.cust.local "/bzt-B" Time: 27.11.2016 16:32:07
    Starting OB2BAR Backup: sbztnv0818.cust.local:/4/sbztnv0818.cust.local/50213439-1ce7-76d1-ac44-3ff8adf7071c/6000C29f-c273-0767-0ae8-5ab495d06c1f "VEAgentDisk"

    [Normal] From: VEPALIB_VMWARE@dpma04.cust.local "/bzt-B" Time: 27.11.2016 16:32:07
    Virtual Machine 'theGuard_SMC': Backing up ...
    Disk: scsi0:0
    Transport method: NBD
    Disk buffer: 1 MB

    [Normal] From: OB2BAR_VEPA_BAR@dpma04.cust.local "/bzt-B" Time: 27.11.2016 16:32:07
    Starting OB2BAR Backup: sbztnv0818.cust.local:/4/sbztnv0818.cust.local/50187015-41ac-70dc-f6a6-19492f30e146/6000C29e-22d8-2db3-2690-35bb6634caee "VEAgentDisk"

    [Normal] From: VEPALIB_VMWARE@dpma04.cust.local "/bzt-B" Time: 27.11.2016 16:32:07
    Virtual Machine 'theGuard_SMC': Disk 'scsi0:0' datastore information
    Datastore: P2_02_bzt
    Datastore type: VMFS
    Datastore Storage WWN: 600c0ff0001dc3481c543c5701000000

    [Normal] From: VEPALIB_VMWARE@dpma04.cust.local "/bzt-B" Time: 27.11.2016 16:32:07
    Virtual Machine 'Facility-Mgm': Backing up ...
    Disk: scsi0:0
    Transport method: NBD
    Disk buffer: 1 MB

    [Normal] From: VEPALIB_VMWARE@dpma04.cust.local "/bzt-B" Time: 27.11.2016 16:32:07
    Virtual Machine 'Facility-Mgm': Disk 'scsi0:0' datastore information
    Datastore: P2_02_bzt
    Datastore type: VMFS
    Datastore Storage WWN: 600c0ff0001dc3481c543c5701000000

    [Normal] From: VEPALIB_VMWARE@dpma04.cust.local "/bzt-B" Time: 27.11.2016 16:32:13
    Virtual Machine 'AV_ID_ENGINE_02' (UUID '5018997a-3ce8-54e5-ff80-49e5d5a4f8cd', Disk 'scsi0:0'): Starting full backup ... (skipping lines)

     

    [Normal] From: OB2BAR_VEPA_BAR@dpma04.cust.local "/bzt-B" Time: 27.11.2016 20:45:03
    Starting OB2BAR Backup: sbztnv0818.cust.local:/4/sbztnv0818.cust.local/52fc82f0-b287-8ff0-1a3d-89427e7a49da "VEAgent"

    [Normal] From: BMA@dpma04.cust.local "B2D_bzt_VM_gw-dpma04 [GW 6384:0:3941272586470069107]" Time: 27.11.2016 20:45:03
    Unloading medium to slot \\dpma04.cust.local\B2D_bzt_VM\c80010ac_583b1b5a_0858_3a3f from device B2D_bzt_VM_gw-dpma04 [GW 6384:0:3941272586470069107]

    [Normal] From: BMA@dpma04.cust.local "B2D_bzt_VM_gw-dpma04 [GW 6384:0:3941272586470069107]" Time: 27.11.2016 20:45:03
    Loading medium from slot \\dpma04.cust.local\B2D_bzt_VM\c80010ac_583b37bf_0858_3a85 to device B2D_bzt_VM_gw-dpma04 [GW 6384:0:3941272586470069107]

    [Normal] From: BSM@dpcm01.cust.local "sbztnv0818_Infrastruktur_B2D" Time: 27.11.2016 20:45:03

    Server-side Deduplication Statistics for sbztnv0818.cust.local:/4/sbztnv0818.cust.local/52fc82f0-b287-8ff0-1a3d-89427e7a49da "VEAgent".
    Using device: "B2D_bzt_VM_gw-dpma04 [GW 6384:0:3941272586470069107]@dpma04.cust.local":
    Mbytes Total: ................. 282053 MB
    Mbytes Written to Disk: ....... 5587 MB
    Deduplication Ratio: .......... 50.4 : 1

    [Normal] From: OB2BAR_VEPA_BAR@dpma04.cust.local "/bzt-B" Time: 27.11.2016 20:45:05
    Completed OB2BAR Backup: sbztnv0818.cust.local:/4/sbztnv0818.cust.local/52fc82f0-b287-8ff0-1a3d-89427e7a49da "VEAgent"

    [Normal] From: VEPALIB_VMWARE@dpma04.cust.local "/bzt-B" Time: 27.11.2016 20:45:06
    Virtual Machine 'Management': Removing snapshot ...

    ...

    [Normal] From: VEPALIB_VMWARE@dpma04.cust.local "/bzt-B" Time: 27.11.2016 20:45:58
    Virtual Machine 'SbztNV0868.cust.local': Unlocking vMotion ...

    [Normal] From: BSM@dpcm01.cust.local "sbztnv0818_Infrastruktur_B2D" Time: 27.11.2016 20:47:57
    OB2BAR application on "dpma04.cust.local" disconnected.

    [Normal] From: BMA@dpma04.cust.local "B2D_bzt_VM_gw-dpma04 [GW 6384:0:3941272586470069107]" Time: 27.11.2016 20:47:57
    Unloading medium to slot \\dpma04.cust.local\B2D_bzt_VM\c80010ac_583b37bf_0858_3a85 from device B2D_bzt_VM_gw-dpma04 [GW 6384:0:3941272586470069107]

    [Normal] From: BMA@dpma04.cust.local "B2D_bzt_VM_gw-dpma04 [GW 6384:0:3941272586470069107]" Time: 27.11.2016 20:47:57
    COMPLETED Media Agent "B2D_bzt_VM_gw-dpma04 [GW 6384:0:3941272586470069107]"

    [Normal] From: BMA@dpma03.cust.local "B2D_bzt_VM_gw-dpma03 [GW 6384:1:3941267088911930483]" Time: 27.11.2016 20:47:57
    Unloading medium to slot \\dpma04.cust.local\B2D_bzt_VM\c80010ac_583b0ff0_0858_3a27 from device B2D_bzt_VM_gw-dpma03 [GW 6384:1:3941267088911930483]

    [Normal] From: BMA@dpma03.cust.local "B2D_bzt_VM_gw-dpma03 [GW 6384:1:3941267088911930483]" Time: 27.11.2016 20:47:57
    COMPLETED Media Agent "B2D_bzt_VM_gw-dpma03 [GW 6384:1:3941267088911930483]"

    [Normal] From: BMA@dpma04.cust.local "B2D_bzt_VM_gw-dpma04 [GW 6384:1:3941272586470069107]" Time: 27.11.2016 20:47:58
    Unloading medium to slot \\dpma04.cust.local\B2D_bzt_VM\c80010ac_583b3737_0858_3a81 from device B2D_bzt_VM_gw-dpma04 [GW 6384:1:3941272586470069107]

    [Normal] From: BMA@dpma04.cust.local "B2D_bzt_VM_gw-dpma04 [GW 6384:1:3941272586470069107]" Time: 27.11.2016 20:47:58
    COMPLETED Media Agent "B2D_bzt_VM_gw-dpma04 [GW 6384:1:3941272586470069107]"

    [Normal] From: BSM@dpcm01.cust.local "sbztnv0818_Infrastruktur_B2D" Time: 27.11.2016 20:47:58

    Backup Statistics:

    Session Queuing Time (hours) 0,00
    -------------------------------------------
    Completed Disk Agents ........ 35
    Failed Disk Agents ........... 0
    Aborted Disk Agents .......... 0
    -------------------------------------------
    Disk Agents Total ........... 35
    ===========================================
    Completed Media Agents ....... 6
    Failed Media Agents .......... 0
    Aborted Media Agents ......... 0
    -------------------------------------------
    Media Agents Total .......... 6
    ===========================================
    Throttled Gateways ........... 0
    Gateways Total ............... 2
    ===========================================
    Overall Deduplication Ratio .. 130.2 : 1
    ===========================================
    Mbytes Total ................. 1406302 MB
    Used Media Total ............. 35
    Disk Agent Errors Total ...... 0

     

     

    best regards

    Joerg

  • Joerg,

    I think you can attach the report instead of pasting it. I see 6 media agents used in the end of session report, while I see only 2 starting in the messages you pasted. Please also attach the device details (omnidownload).

    Koen

  • And the "Loading" and"Unloading" messages don't seem to correspond to each other ....

  • Must have removed to much. :-) Since I can't upload txt-files I just renamend them to gif.

    I've copied the library/device infos in one file for each store.

    Checked the "STARTING Media Agent"/"COMPLETED Media Agent"-messages: 6 hits both, so should be OK now.

    Kind regards

    Joerg

  • Hi Joerg,

    do you use global variable LimitInitGatewayExpansion? There were some problem around it in 9.06 

  • Hi,

    can't find LimitInitGatewayExpansion in the global-file.

    Best regards
    Joerg

  • Verified Answer

    I could reproduce this with: 


    C:\Users\administrator>csm -ver
    HPE Data Protector A.09.06: CSM, internal build 107, built on Tuesday, April 5,
    2016, 12:07 AM

    It seems to be caused by this change request: 

    Change Request: QCCR2A65552:
      Symptom:
        The Data Protector object copy session allocates too much SO connections and
        fails with the following error message:
        |Warning| From: CSM@<hostname> "" Time: <Date><Time>
        There are no available connections on store '' for the device named ''.
      Resolution:
        The csm.exe file is modified to correctly handle the Data Protector object
        copy session.


    CSM is expanding as many source gw as there is concurency for target devices used in the copy session. And with another storeonce target that is always one per gateway. 

    This was later on fixed in 9.07 and higher versions. 

     

  • OK, I'll give it a try and will schedule an update to 9.08.

    best regards

    Joerg

Reply Children
No Data